2024-09-21T12:15:43.355 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-21T12:15:43.372 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-21T12:15:43.452 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/gabrioux-2024-09-21_11:23:40-orch:cephadm-wip-guits-main-2024-09-21-0842-distro-default-smithi/7913934 branch: wip-guits-main-2024-09-21-0842 description: orch:cephadm/upgrade/{1-start-distro/1-start-centos_9.stream-reef 2-repo_digest/defaut 3-upgrade/simple 4-wait 5-upgrade-ls agent/on mon_election/classic} email: gabrioux@ibm.com first_in_suite: false job_id: '7913934' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: gabrioux-2024-09-21_11:23:40-orch:cephadm-wip-guits-main-2024-09-21-0842-distro-default-smithi no_nested_subset: false os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-guits-main-2024-09-21-0842 ceph: conf: global: mon election default strategy: 1 mgr: debug mgr: 20 debug ms: 1 mgr/cephadm/use_agent: true mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - CEPHADM_STRAY_DAEMON - CEPHADM_FAILED_DAEMON - CEPHADM_AGENT_DOWN log-only-match: - CEPHADM_ sha1: 441784946648ff53dd310293f9dd752afeeea72a ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 441784946648ff53dd310293f9dd752afeeea72a workunit: branch: wip-guits-main-2024-09-21-0842 sha1: 441784946648ff53dd310293f9dd752afeeea72a owner: scheduled_gabrioux@teuthology priority: 70 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - grafana.a - node-exporter.b seed: 5896 sha1: 441784946648ff53dd310293f9dd752afeeea72a sleep_before_teardown: 0 subset: 1/8 suite: orch:cephadm suite_branch: wip-guits-main-2024-09-21-0842 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_441784946648ff53dd310293f9dd752afeeea72a/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 441784946648ff53dd310293f9dd752afeeea72a targets: smithi079.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBC3a6ijB4M5Tv3sR+05Z+FlquVe730MdvDg/yQQzjh0RIwXkfIMy9XfURYJ/lTmAK1b1QWV27ueaA77Mmmf68Ps= smithi104.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLqT+XmCiyQF0Kj+jCdl3USGze6N3zm5+JA0xy0LRZp/w2UUByQEMCzHdKb8nnE1hXT90K+k8Act7R25uEuqTjw= tasks: - cephadm: compiled_cephadm_branch: reef image: quay.ceph.io/ceph-ci/ceph:reef - cephadm.shell: env: - sha1 mon.a: - radosgw-admin realm create --rgw-realm=r --default - radosgw-admin zonegroup create --rgw-zonegroup=default --master --default - radosgw-admin zone create --rgw-zonegroup=default --rgw-zone=z --master --default - radosgw-admin period update --rgw-realm=r --commit - ceph orch apply rgw foo --realm r --zone z --placement=2 --port=8000 - ceph orch apply rgw smpl - ceph osd pool create foo - rbd pool init foo - ceph orch apply iscsi foo u p - sleep 120 - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 mon.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done - ceph orch ps - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph orch upgrade status - ceph health detail - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 - ceph orch ls | grep '^osd ' - cephadm.shell: mon.a: - ceph orch upgrade ls - ceph orch upgrade ls --image quay.io/ceph/ceph --show-all-versions | grep 16.2.0 - ceph orch upgrade ls --image quay.io/ceph/ceph --tags | grep v16.2.2 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-21_11:23:40 tube: smithi user: gabrioux verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2902 2024-09-21T12:15:43.453 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_441784946648ff53dd310293f9dd752afeeea72a/qa; will attempt to use it 2024-09-21T12:15:43.453 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_441784946648ff53dd310293f9dd752afeeea72a/qa/tasks 2024-09-21T12:15:43.453 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-21T12:15:43.455 INFO:teuthology.task.internal:Checking packages... 2024-09-21T12:15:43.478 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '441784946648ff53dd310293f9dd752afeeea72a' 2024-09-21T12:15:43.478 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-21T12:15:43.478 INFO:teuthology.packaging:ref: None 2024-09-21T12:15:43.478 INFO:teuthology.packaging:tag: None 2024-09-21T12:15:43.478 INFO:teuthology.packaging:branch: wip-guits-main-2024-09-21-0842 2024-09-21T12:15:43.478 INFO:teuthology.packaging:sha1: 441784946648ff53dd310293f9dd752afeeea72a 2024-09-21T12:15:43.479 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-guits-main-2024-09-21-0842 2024-09-21T12:15:43.629 INFO:teuthology.task.internal:Found packages for ceph version 19.3.0-5104.g44178494 2024-09-21T12:15:43.631 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-21T12:15:43.635 INFO:teuthology.task.internal:no buildpackages task found 2024-09-21T12:15:43.635 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-21T12:15:43.639 INFO:teuthology.task.internal:Saving configuration 2024-09-21T12:15:43.651 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-21T12:15:43.657 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-21T12:15:43.684 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi079.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/gabrioux-2024-09-21_11:23:40-orch:cephadm-wip-guits-main-2024-09-21-0842-distro-default-smithi/7913934', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-21 12:10:49.042392', 'locked_by': 'scheduled_gabrioux@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBC3a6ijB4M5Tv3sR+05Z+FlquVe730MdvDg/yQQzjh0RIwXkfIMy9XfURYJ/lTmAK1b1QWV27ueaA77Mmmf68Ps='} 2024-09-21T12:15:43.704 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi104.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/gabrioux-2024-09-21_11:23:40-orch:cephadm-wip-guits-main-2024-09-21-0842-distro-default-smithi/7913934', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-21 12:10:49.040722', 'locked_by': 'scheduled_gabrioux@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLqT+XmCiyQF0Kj+jCdl3USGze6N3zm5+JA0xy0LRZp/w2UUByQEMCzHdKb8nnE1hXT90K+k8Act7R25uEuqTjw='} 2024-09-21T12:15:43.704 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-21T12:15:43.709 INFO:teuthology.task.internal:roles: ubuntu@smithi079.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a', 'alertmanager.a'] 2024-09-21T12:15:43.710 INFO:teuthology.task.internal:roles: ubuntu@smithi104.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b'] 2024-09-21T12:15:43.710 INFO:teuthology.run_tasks:Running task console_log... 2024-09-21T12:15:43.763 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f6d79410280>, signals=[15]) 2024-09-21T12:15:43.763 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-21T12:15:43.768 INFO:teuthology.task.internal:Opening connections... 2024-09-21T12:15:43.768 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi079.front.sepia.ceph.com 2024-09-21T12:15:43.769 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi079.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T12:15:43.843 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi104.front.sepia.ceph.com 2024-09-21T12:15:43.844 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T12:15:43.918 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-21T12:15:43.924 DEBUG:teuthology.orchestra.run.smithi079:> uname -m 2024-09-21T12:15:43.941 INFO:teuthology.orchestra.run.smithi079.stdout:x86_64 2024-09-21T12:15:43.941 DEBUG:teuthology.orchestra.run.smithi079:> cat /etc/os-release 2024-09-21T12:15:43.998 INFO:teuthology.orchestra.run.smithi079.stdout:NAME="CentOS Stream" 2024-09-21T12:15:43.998 INFO:teuthology.orchestra.run.smithi079.stdout:VERSION="9" 2024-09-21T12:15:43.998 INFO:teuthology.orchestra.run.smithi079.stdout:ID="centos" 2024-09-21T12:15:43.998 INFO:teuthology.orchestra.run.smithi079.stdout:ID_LIKE="rhel fedora" 2024-09-21T12:15:43.998 INFO:teuthology.orchestra.run.smithi079.stdout:VERSION_ID="9" 2024-09-21T12:15:43.998 INFO:teuthology.orchestra.run.smithi079.stdout:PLATFORM_ID="platform:el9" 2024-09-21T12:15:43.998 INFO:teuthology.orchestra.run.smithi079.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-21T12:15:43.999 INFO:teuthology.orchestra.run.smithi079.stdout:ANSI_COLOR="0;31" 2024-09-21T12:15:43.999 INFO:teuthology.orchestra.run.smithi079.stdout:LOGO="fedora-logo-icon" 2024-09-21T12:15:43.999 INFO:teuthology.orchestra.run.smithi079.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-21T12:15:43.999 INFO:teuthology.orchestra.run.smithi079.stdout:HOME_URL="https://centos.org/" 2024-09-21T12:15:43.999 INFO:teuthology.orchestra.run.smithi079.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-21T12:15:43.999 INFO:teuthology.orchestra.run.smithi079.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-21T12:15:43.999 INFO:teuthology.orchestra.run.smithi079.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-21T12:15:44.000 INFO:teuthology.lock.ops:Updating smithi079.front.sepia.ceph.com on lock server 2024-09-21T12:15:44.029 DEBUG:teuthology.orchestra.run.smithi104:> uname -m 2024-09-21T12:15:44.048 INFO:teuthology.orchestra.run.smithi104.stdout:x86_64 2024-09-21T12:15:44.048 DEBUG:teuthology.orchestra.run.smithi104:> cat /etc/os-release 2024-09-21T12:15:44.106 INFO:teuthology.orchestra.run.smithi104.stdout:NAME="CentOS Stream" 2024-09-21T12:15:44.106 INFO:teuthology.orchestra.run.smithi104.stdout:VERSION="9" 2024-09-21T12:15:44.106 INFO:teuthology.orchestra.run.smithi104.stdout:ID="centos" 2024-09-21T12:15:44.106 INFO:teuthology.orchestra.run.smithi104.stdout:ID_LIKE="rhel fedora" 2024-09-21T12:15:44.107 INFO:teuthology.orchestra.run.smithi104.stdout:VERSION_ID="9" 2024-09-21T12:15:44.107 INFO:teuthology.orchestra.run.smithi104.stdout:PLATFORM_ID="platform:el9" 2024-09-21T12:15:44.107 INFO:teuthology.orchestra.run.smithi104.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-21T12:15:44.107 INFO:teuthology.orchestra.run.smithi104.stdout:ANSI_COLOR="0;31" 2024-09-21T12:15:44.107 INFO:teuthology.orchestra.run.smithi104.stdout:LOGO="fedora-logo-icon" 2024-09-21T12:15:44.107 INFO:teuthology.orchestra.run.smithi104.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-21T12:15:44.107 INFO:teuthology.orchestra.run.smithi104.stdout:HOME_URL="https://centos.org/" 2024-09-21T12:15:44.107 INFO:teuthology.orchestra.run.smithi104.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-21T12:15:44.107 INFO:teuthology.orchestra.run.smithi104.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-21T12:15:44.107 INFO:teuthology.orchestra.run.smithi104.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-21T12:15:44.108 INFO:teuthology.lock.ops:Updating smithi104.front.sepia.ceph.com on lock server 2024-09-21T12:15:44.137 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-21T12:15:44.146 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-21T12:15:44.173 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-21T12:15:44.174 DEBUG:teuthology.orchestra.run.smithi079:> test '!' -e /home/ubuntu/cephtest 2024-09-21T12:15:44.176 DEBUG:teuthology.orchestra.run.smithi104:> test '!' -e /home/ubuntu/cephtest 2024-09-21T12:15:44.193 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-21T12:15:44.199 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-21T12:15:44.199 DEBUG:teuthology.orchestra.run.smithi079:> test -z $(ls -A /var/lib/ceph) 2024-09-21T12:15:44.234 DEBUG:teuthology.orchestra.run.smithi104:> test -z $(ls -A /var/lib/ceph) 2024-09-21T12:15:44.361 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-21T12:15:44.417 INFO:teuthology.run_tasks:Running task kernel... 2024-09-21T12:15:44.431 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-21T12:15:44.431 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-21T12:15:44.431 DEBUG:teuthology.orchestra.run.smithi079:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-21T12:15:44.432 DEBUG:teuthology.orchestra.run.smithi104:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-21T12:15:44.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:15:44.448 DEBUG:teuthology.orchestra.run.smithi079:> uname -r 2024-09-21T12:15:44.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:15:44.450 DEBUG:teuthology.orchestra.run.smithi104:> uname -r 2024-09-21T12:15:44.505 INFO:teuthology.orchestra.run.smithi079.stdout:5.14.0-503.el9.x86_64 2024-09-21T12:15:44.505 INFO:teuthology.task.kernel:Running kernel on smithi079: 5.14.0-503.el9.x86_64 2024-09-21T12:15:44.506 DEBUG:teuthology.orchestra.run.smithi079:> sudo yum install -y kernel 2024-09-21T12:15:44.507 INFO:teuthology.orchestra.run.smithi104.stdout:5.14.0-503.el9.x86_64 2024-09-21T12:15:44.508 INFO:teuthology.task.kernel:Running kernel on smithi104: 5.14.0-503.el9.x86_64 2024-09-21T12:15:44.508 DEBUG:teuthology.orchestra.run.smithi104:> sudo yum install -y kernel 2024-09-21T12:15:46.769 INFO:teuthology.orchestra.run.smithi079.stdout:CentOS Stream 9 - BaseOS 16 MB/s | 8.3 MB 00:00 2024-09-21T12:15:46.937 INFO:teuthology.orchestra.run.smithi104.stdout:CentOS Stream 9 - BaseOS 13 MB/s | 8.3 MB 00:00 2024-09-21T12:15:48.913 INFO:teuthology.orchestra.run.smithi104.stdout:CentOS Stream 9 - AppStream 22 MB/s | 20 MB 00:00 2024-09-21T12:15:53.113 INFO:teuthology.orchestra.run.smithi104.stdout:CentOS Stream 9 - CRB 17 MB/s | 6.5 MB 00:00 2024-09-21T12:15:54.530 INFO:teuthology.orchestra.run.smithi104.stdout:CentOS Stream 9 - Extras packages 89 kB/s | 19 kB 00:00 2024-09-21T12:15:55.634 INFO:teuthology.orchestra.run.smithi104.stdout:Extra Packages for Enterprise Linux 23 MB/s | 23 MB 00:00 2024-09-21T12:15:55.978 INFO:teuthology.orchestra.run.smithi079.stdout:CentOS Stream 9 - AppStream 2.5 MB/s | 20 MB 00:08 2024-09-21T12:16:00.965 INFO:teuthology.orchestra.run.smithi104.stdout:lab-extras 50 kB/s | 1.7 kB 00:00 2024-09-21T12:16:01.053 INFO:teuthology.orchestra.run.smithi079.stdout:CentOS Stream 9 - CRB 5.4 MB/s | 6.5 MB 00:01 2024-09-21T12:16:02.478 INFO:teuthology.orchestra.run.smithi079.stdout:CentOS Stream 9 - Extras packages 90 kB/s | 19 kB 00:00 2024-09-21T12:16:02.813 INFO:teuthology.orchestra.run.smithi104.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-21T12:16:02.813 INFO:teuthology.orchestra.run.smithi104.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-21T12:16:02.923 INFO:teuthology.orchestra.run.smithi104.stdout:Dependencies resolved. 2024-09-21T12:16:02.928 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-09-21T12:16:02.928 INFO:teuthology.orchestra.run.smithi104.stdout: Package Architecture Version Repository Size 2024-09-21T12:16:02.928 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-09-21T12:16:02.928 INFO:teuthology.orchestra.run.smithi104.stdout:Installing: 2024-09-21T12:16:02.928 INFO:teuthology.orchestra.run.smithi104.stdout: kernel x86_64 5.14.0-508.el9 baseos 36 k 2024-09-21T12:16:02.928 INFO:teuthology.orchestra.run.smithi104.stdout:Installing dependencies: 2024-09-21T12:16:02.928 INFO:teuthology.orchestra.run.smithi104.stdout: kernel-core x86_64 5.14.0-508.el9 baseos 16 M 2024-09-21T12:16:02.928 INFO:teuthology.orchestra.run.smithi104.stdout: kernel-modules x86_64 5.14.0-508.el9 baseos 35 M 2024-09-21T12:16:02.928 INFO:teuthology.orchestra.run.smithi104.stdout: kernel-modules-core x86_64 5.14.0-508.el9 baseos 29 M 2024-09-21T12:16:02.928 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-21T12:16:02.929 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction Summary 2024-09-21T12:16:02.929 INFO:teuthology.orchestra.run.smithi104.stdout:================================================================================ 2024-09-21T12:16:02.929 INFO:teuthology.orchestra.run.smithi104.stdout:Install 4 Packages 2024-09-21T12:16:02.929 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-21T12:16:02.929 INFO:teuthology.orchestra.run.smithi104.stdout:Total download size: 79 M 2024-09-21T12:16:02.929 INFO:teuthology.orchestra.run.smithi104.stdout:Installed size: 126 M 2024-09-21T12:16:02.930 INFO:teuthology.orchestra.run.smithi104.stdout:Downloading Packages: 2024-09-21T12:16:03.275 INFO:teuthology.orchestra.run.smithi104.stdout:(1/4): kernel-5.14.0-508.el9.x86_64.rpm 409 kB/s | 36 kB 00:00 2024-09-21T12:16:03.632 INFO:teuthology.orchestra.run.smithi079.stdout:Extra Packages for Enterprise Linux 22 MB/s | 23 MB 00:01 2024-09-21T12:16:03.816 INFO:teuthology.orchestra.run.smithi104.stdout:(2/4): kernel-modules-5.14.0-508.el9.x86_64.rpm 55 MB/s | 35 MB 00:00 2024-09-21T12:16:03.958 INFO:teuthology.orchestra.run.smithi104.stdout:(3/4): kernel-core-5.14.0-508.el9.x86_64.rpm 20 MB/s | 16 MB 00:00 2024-09-21T12:16:04.209 INFO:teuthology.orchestra.run.smithi104.stdout:(4/4): kernel-modules-core-5.14.0-508.el9.x86_6 31 MB/s | 29 MB 00:00 2024-09-21T12:16:04.209 INFO:teuthology.orchestra.run.smithi104.stdout:-------------------------------------------------------------------------------- 2024-09-21T12:16:04.209 INFO:teuthology.orchestra.run.smithi104.stdout:Total 62 MB/s | 79 MB 00:01 2024-09-21T12:16:04.895 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction check 2024-09-21T12:16:05.221 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction check succeeded. 2024-09-21T12:16:05.221 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction test 2024-09-21T12:16:06.210 INFO:teuthology.orchestra.run.smithi104.stdout:Transaction test succeeded. 2024-09-21T12:16:06.211 INFO:teuthology.orchestra.run.smithi104.stdout:Running transaction 2024-09-21T12:16:08.664 INFO:teuthology.orchestra.run.smithi104.stdout: Preparing : 1/1 2024-09-21T12:16:08.979 INFO:teuthology.orchestra.run.smithi079.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-09-21T12:16:09.291 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : kernel-modules-core-5.14.0-508.el9.x86_64 1/4 2024-09-21T12:16:09.374 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-21T12:16:10.894 INFO:teuthology.orchestra.run.smithi079.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-21T12:16:10.894 INFO:teuthology.orchestra.run.smithi079.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-21T12:16:10.979 INFO:teuthology.orchestra.run.smithi079.stdout:Dependencies resolved. 2024-09-21T12:16:10.984 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-09-21T12:16:10.985 INFO:teuthology.orchestra.run.smithi079.stdout: Package Architecture Version Repository Size 2024-09-21T12:16:10.985 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-09-21T12:16:10.985 INFO:teuthology.orchestra.run.smithi079.stdout:Installing: 2024-09-21T12:16:10.985 INFO:teuthology.orchestra.run.smithi079.stdout: kernel x86_64 5.14.0-508.el9 baseos 36 k 2024-09-21T12:16:10.985 INFO:teuthology.orchestra.run.smithi079.stdout:Installing dependencies: 2024-09-21T12:16:10.985 INFO:teuthology.orchestra.run.smithi079.stdout: kernel-core x86_64 5.14.0-508.el9 baseos 16 M 2024-09-21T12:16:10.986 INFO:teuthology.orchestra.run.smithi079.stdout: kernel-modules x86_64 5.14.0-508.el9 baseos 35 M 2024-09-21T12:16:10.986 INFO:teuthology.orchestra.run.smithi079.stdout: kernel-modules-core x86_64 5.14.0-508.el9 baseos 29 M 2024-09-21T12:16:10.986 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-21T12:16:10.986 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction Summary 2024-09-21T12:16:10.986 INFO:teuthology.orchestra.run.smithi079.stdout:================================================================================ 2024-09-21T12:16:10.986 INFO:teuthology.orchestra.run.smithi079.stdout:Install 4 Packages 2024-09-21T12:16:10.987 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-21T12:16:10.987 INFO:teuthology.orchestra.run.smithi079.stdout:Total download size: 79 M 2024-09-21T12:16:10.988 INFO:teuthology.orchestra.run.smithi079.stdout:Installed size: 126 M 2024-09-21T12:16:10.988 INFO:teuthology.orchestra.run.smithi079.stdout:Downloading Packages: 2024-09-21T12:16:11.097 INFO:teuthology.orchestra.run.smithi079.stdout:(1/4): kernel-5.14.0-508.el9.x86_64.rpm 397 kB/s | 36 kB 00:00 2024-09-21T12:16:11.481 INFO:teuthology.orchestra.run.smithi079.stdout:(2/4): kernel-core-5.14.0-508.el9.x86_64.rpm 33 MB/s | 16 MB 00:00 2024-09-21T12:16:11.915 INFO:teuthology.orchestra.run.smithi079.stdout:(3/4): kernel-modules-5.14.0-508.el9.x86_64.rpm 38 MB/s | 35 MB 00:00 2024-09-21T12:16:12.152 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-21T12:16:12.231 INFO:teuthology.orchestra.run.smithi079.stdout:(4/4): kernel-modules-core-5.14.0-508.el9.x86_6 25 MB/s | 29 MB 00:01 2024-09-21T12:16:12.232 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-09-21T12:16:12.232 INFO:teuthology.orchestra.run.smithi079.stdout:Total 63 MB/s | 79 MB 00:01 2024-09-21T12:16:12.285 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-21T12:16:12.910 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction check 2024-09-21T12:16:13.259 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction check succeeded. 2024-09-21T12:16:13.259 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction test 2024-09-21T12:16:14.252 INFO:teuthology.orchestra.run.smithi079.stdout:Transaction test succeeded. 2024-09-21T12:16:14.252 INFO:teuthology.orchestra.run.smithi079.stdout:Running transaction 2024-09-21T12:16:16.674 INFO:teuthology.orchestra.run.smithi079.stdout: Preparing : 1/1 2024-09-21T12:16:17.119 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-21T12:16:17.260 INFO:teuthology.orchestra.run.smithi104.stdout: Installing : kernel-5.14.0-508.el9.x86_64 4/4 2024-09-21T12:16:17.334 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : kernel-modules-core-5.14.0-508.el9.x86_64 1/4 2024-09-21T12:16:17.415 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-21T12:16:20.177 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-21T12:16:20.301 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-21T12:16:22.113 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-21T12:16:25.139 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-21T12:16:25.343 INFO:teuthology.orchestra.run.smithi079.stdout: Installing : kernel-5.14.0-508.el9.x86_64 4/4 2024-09-21T12:16:30.219 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-21T12:17:00.368 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 4/4 2024-09-21T12:17:00.408 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 4/4 2024-09-21T12:17:00.785 INFO:teuthology.orchestra.run.smithi104.stdout: Running scriptlet: kernel-5.14.0-508.el9.x86_64 4/4 2024-09-21T12:17:00.785 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : kernel-5.14.0-508.el9.x86_64 1/4 2024-09-21T12:17:00.785 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-21T12:17:00.785 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-21T12:17:01.617 INFO:teuthology.orchestra.run.smithi104.stdout: Verifying : kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-21T12:17:01.617 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-21T12:17:01.618 INFO:teuthology.orchestra.run.smithi104.stdout:Installed: 2024-09-21T12:17:01.618 INFO:teuthology.orchestra.run.smithi104.stdout: kernel-5.14.0-508.el9.x86_64 2024-09-21T12:17:01.618 INFO:teuthology.orchestra.run.smithi104.stdout: kernel-core-5.14.0-508.el9.x86_64 2024-09-21T12:17:01.618 INFO:teuthology.orchestra.run.smithi104.stdout: kernel-modules-5.14.0-508.el9.x86_64 2024-09-21T12:17:01.618 INFO:teuthology.orchestra.run.smithi104.stdout: kernel-modules-core-5.14.0-508.el9.x86_64 2024-09-21T12:17:01.618 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-21T12:17:01.618 INFO:teuthology.orchestra.run.smithi104.stdout:Complete! 2024-09-21T12:17:02.018 DEBUG:teuthology.orchestra.run.smithi104:> rpm -q kernel | sort -rV | head -n 1 2024-09-21T12:17:02.100 INFO:teuthology.orchestra.run.smithi104.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-21T12:17:02.100 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-21T12:17:02.100 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-508.el9.x86_64 2024-09-21T12:17:02.100 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-21T12:17:02.101 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-21T12:17:02.101 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-09-21T12:17:02.101 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi104.front.sepia.ceph.com, path=None, version=distro) 2024-09-21T12:17:02.101 DEBUG:teuthology.orchestra.run.smithi104:> rpm -q kernel | sort -rV | head -n 1 2024-09-21T12:17:02.159 INFO:teuthology.orchestra.run.smithi104.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-21T12:17:02.160 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-21T12:17:02.160 DEBUG:teuthology.orchestra.run.smithi104:> sudo rpm -qi grub2-tools 2024-09-21T12:17:02.192 INFO:teuthology.orchestra.run.smithi104.stdout:Name : grub2-tools 2024-09-21T12:17:02.192 INFO:teuthology.orchestra.run.smithi104.stdout:Epoch : 1 2024-09-21T12:17:02.192 INFO:teuthology.orchestra.run.smithi104.stdout:Version : 2.06 2024-09-21T12:17:02.192 INFO:teuthology.orchestra.run.smithi104.stdout:Release : 82.el9 2024-09-21T12:17:02.192 INFO:teuthology.orchestra.run.smithi104.stdout:Architecture: x86_64 2024-09-21T12:17:02.192 INFO:teuthology.orchestra.run.smithi104.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-21T12:17:02.192 INFO:teuthology.orchestra.run.smithi104.stdout:Group : Unspecified 2024-09-21T12:17:02.193 INFO:teuthology.orchestra.run.smithi104.stdout:Size : 8274670 2024-09-21T12:17:02.193 INFO:teuthology.orchestra.run.smithi104.stdout:License : GPLv3+ 2024-09-21T12:17:02.193 INFO:teuthology.orchestra.run.smithi104.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-21T12:17:02.193 INFO:teuthology.orchestra.run.smithi104.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-21T12:17:02.193 INFO:teuthology.orchestra.run.smithi104.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-21T12:17:02.193 INFO:teuthology.orchestra.run.smithi104.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-21T12:17:02.193 INFO:teuthology.orchestra.run.smithi104.stdout:Packager : builder@centos.org 2024-09-21T12:17:02.193 INFO:teuthology.orchestra.run.smithi104.stdout:Vendor : CentOS 2024-09-21T12:17:02.193 INFO:teuthology.orchestra.run.smithi104.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-21T12:17:02.193 INFO:teuthology.orchestra.run.smithi104.stdout:Summary : Support tools for GRUB. 2024-09-21T12:17:02.193 INFO:teuthology.orchestra.run.smithi104.stdout:Description : 2024-09-21T12:17:02.193 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-21T12:17:02.193 INFO:teuthology.orchestra.run.smithi104.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-21T12:17:02.193 INFO:teuthology.orchestra.run.smithi104.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-21T12:17:02.193 INFO:teuthology.orchestra.run.smithi104.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-21T12:17:02.194 INFO:teuthology.orchestra.run.smithi104.stdout:hardware devices. 2024-09-21T12:17:02.194 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-21T12:17:02.194 INFO:teuthology.orchestra.run.smithi104.stdout:This subpackage provides tools for support of all platforms. 2024-09-21T12:17:02.194 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-21T12:17:02.195 INFO:teuthology.task.kernel:Updating grub on smithi104 to boot 5.14.0-508.el9.x86_64 2024-09-21T12:17:02.195 DEBUG:teuthology.orchestra.run.smithi104:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-21T12:17:02.798 INFO:teuthology.orchestra.run.smithi104.stderr:Generating grub configuration file ... 2024-09-21T12:17:04.188 INFO:teuthology.orchestra.run.smithi104.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-21T12:17:04.224 INFO:teuthology.orchestra.run.smithi104.stderr:done 2024-09-21T12:17:04.226 DEBUG:teuthology.orchestra.run.smithi104:> mktemp 2024-09-21T12:17:04.241 INFO:teuthology.orchestra.run.smithi104.stdout:/tmp/tmp.0i3EKMtIdj 2024-09-21T12:17:04.242 DEBUG:teuthology.orchestra.run.smithi104:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.0i3EKMtIdj 2024-09-21T12:17:04.307 DEBUG:teuthology.orchestra.run.smithi104:> sudo chmod 0666 /tmp/tmp.0i3EKMtIdj 2024-09-21T12:17:04.442 DEBUG:teuthology.orchestra.remote:smithi104:/tmp/tmp.0i3EKMtIdj is 6KB 2024-09-21T12:17:04.462 DEBUG:teuthology.orchestra.run.smithi104:> rm -fr /tmp/tmp.0i3EKMtIdj 2024-09-21T12:17:04.477 DEBUG:teuthology.orchestra.run.smithi104:> sudo /bin/ls /boot/loader/entries || true 2024-09-21T12:17:04.538 INFO:teuthology.orchestra.run.smithi104.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-21T12:17:04.539 INFO:teuthology.orchestra.run.smithi104.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-21T12:17:04.539 INFO:teuthology.orchestra.run.smithi104.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-21T12:17:04.539 INFO:teuthology.orchestra.run.smithi104.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64.conf 2024-09-21T12:17:04.540 DEBUG:teuthology.orchestra.run.smithi104:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64 2024-09-21T12:17:04.689 DEBUG:teuthology.orchestra.run.smithi104:> sudo shutdown -r now 2024-09-21T12:17:08.703 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 4/4 2024-09-21T12:17:08.743 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 4/4 2024-09-21T12:17:09.125 INFO:teuthology.orchestra.run.smithi079.stdout: Running scriptlet: kernel-5.14.0-508.el9.x86_64 4/4 2024-09-21T12:17:09.125 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : kernel-5.14.0-508.el9.x86_64 1/4 2024-09-21T12:17:09.125 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-21T12:17:09.125 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-21T12:17:09.987 INFO:teuthology.orchestra.run.smithi079.stdout: Verifying : kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-21T12:17:09.987 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-21T12:17:09.987 INFO:teuthology.orchestra.run.smithi079.stdout:Installed: 2024-09-21T12:17:09.987 INFO:teuthology.orchestra.run.smithi079.stdout: kernel-5.14.0-508.el9.x86_64 2024-09-21T12:17:09.987 INFO:teuthology.orchestra.run.smithi079.stdout: kernel-core-5.14.0-508.el9.x86_64 2024-09-21T12:17:09.987 INFO:teuthology.orchestra.run.smithi079.stdout: kernel-modules-5.14.0-508.el9.x86_64 2024-09-21T12:17:09.988 INFO:teuthology.orchestra.run.smithi079.stdout: kernel-modules-core-5.14.0-508.el9.x86_64 2024-09-21T12:17:09.988 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-21T12:17:09.988 INFO:teuthology.orchestra.run.smithi079.stdout:Complete! 2024-09-21T12:17:10.354 DEBUG:teuthology.orchestra.run.smithi079:> rpm -q kernel | sort -rV | head -n 1 2024-09-21T12:17:10.455 INFO:teuthology.orchestra.run.smithi079.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-21T12:17:10.455 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-21T12:17:10.455 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-508.el9.x86_64 2024-09-21T12:17:10.455 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-21T12:17:10.455 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-21T12:17:10.455 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-21T12:17:10.455 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi079.front.sepia.ceph.com, path=None, version=distro) 2024-09-21T12:17:10.456 DEBUG:teuthology.orchestra.run.smithi079:> rpm -q kernel | sort -rV | head -n 1 2024-09-21T12:17:10.514 INFO:teuthology.orchestra.run.smithi079.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-21T12:17:10.515 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-21T12:17:10.515 DEBUG:teuthology.orchestra.run.smithi079:> sudo rpm -qi grub2-tools 2024-09-21T12:17:10.555 INFO:teuthology.orchestra.run.smithi079.stdout:Name : grub2-tools 2024-09-21T12:17:10.555 INFO:teuthology.orchestra.run.smithi079.stdout:Epoch : 1 2024-09-21T12:17:10.555 INFO:teuthology.orchestra.run.smithi079.stdout:Version : 2.06 2024-09-21T12:17:10.555 INFO:teuthology.orchestra.run.smithi079.stdout:Release : 82.el9 2024-09-21T12:17:10.555 INFO:teuthology.orchestra.run.smithi079.stdout:Architecture: x86_64 2024-09-21T12:17:10.555 INFO:teuthology.orchestra.run.smithi079.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-21T12:17:10.555 INFO:teuthology.orchestra.run.smithi079.stdout:Group : Unspecified 2024-09-21T12:17:10.555 INFO:teuthology.orchestra.run.smithi079.stdout:Size : 8274670 2024-09-21T12:17:10.555 INFO:teuthology.orchestra.run.smithi079.stdout:License : GPLv3+ 2024-09-21T12:17:10.556 INFO:teuthology.orchestra.run.smithi079.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-21T12:17:10.556 INFO:teuthology.orchestra.run.smithi079.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-21T12:17:10.556 INFO:teuthology.orchestra.run.smithi079.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-21T12:17:10.556 INFO:teuthology.orchestra.run.smithi079.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-21T12:17:10.556 INFO:teuthology.orchestra.run.smithi079.stdout:Packager : builder@centos.org 2024-09-21T12:17:10.556 INFO:teuthology.orchestra.run.smithi079.stdout:Vendor : CentOS 2024-09-21T12:17:10.556 INFO:teuthology.orchestra.run.smithi079.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-21T12:17:10.556 INFO:teuthology.orchestra.run.smithi079.stdout:Summary : Support tools for GRUB. 2024-09-21T12:17:10.556 INFO:teuthology.orchestra.run.smithi079.stdout:Description : 2024-09-21T12:17:10.556 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-21T12:17:10.556 INFO:teuthology.orchestra.run.smithi079.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-21T12:17:10.556 INFO:teuthology.orchestra.run.smithi079.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-21T12:17:10.556 INFO:teuthology.orchestra.run.smithi079.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-21T12:17:10.556 INFO:teuthology.orchestra.run.smithi079.stdout:hardware devices. 2024-09-21T12:17:10.557 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-21T12:17:10.557 INFO:teuthology.orchestra.run.smithi079.stdout:This subpackage provides tools for support of all platforms. 2024-09-21T12:17:10.557 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-21T12:17:10.558 INFO:teuthology.task.kernel:Updating grub on smithi079 to boot 5.14.0-508.el9.x86_64 2024-09-21T12:17:10.558 DEBUG:teuthology.orchestra.run.smithi079:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-21T12:17:11.215 INFO:teuthology.orchestra.run.smithi079.stderr:Generating grub configuration file ... 2024-09-21T12:17:12.444 INFO:teuthology.orchestra.run.smithi079.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-21T12:17:12.475 INFO:teuthology.orchestra.run.smithi079.stderr:done 2024-09-21T12:17:12.478 DEBUG:teuthology.orchestra.run.smithi079:> mktemp 2024-09-21T12:17:12.493 INFO:teuthology.orchestra.run.smithi079.stdout:/tmp/tmp.3hNSxKnLGk 2024-09-21T12:17:12.493 DEBUG:teuthology.orchestra.run.smithi079:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.3hNSxKnLGk 2024-09-21T12:17:12.564 DEBUG:teuthology.orchestra.run.smithi079:> sudo chmod 0666 /tmp/tmp.3hNSxKnLGk 2024-09-21T12:17:12.698 DEBUG:teuthology.orchestra.remote:smithi079:/tmp/tmp.3hNSxKnLGk is 6KB 2024-09-21T12:17:12.747 DEBUG:teuthology.orchestra.run.smithi079:> rm -fr /tmp/tmp.3hNSxKnLGk 2024-09-21T12:17:12.762 DEBUG:teuthology.orchestra.run.smithi079:> sudo /bin/ls /boot/loader/entries || true 2024-09-21T12:17:12.825 INFO:teuthology.orchestra.run.smithi079.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-21T12:17:12.825 INFO:teuthology.orchestra.run.smithi079.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-21T12:17:12.825 INFO:teuthology.orchestra.run.smithi079.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-21T12:17:12.825 INFO:teuthology.orchestra.run.smithi079.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64.conf 2024-09-21T12:17:12.827 DEBUG:teuthology.orchestra.run.smithi079:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64 2024-09-21T12:17:12.991 DEBUG:teuthology.orchestra.run.smithi079:> sudo shutdown -r now 2024-09-21T12:17:34.709 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-09-21T12:17:34.709 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi104.front.sepia.ceph.com' 2024-09-21T12:17:34.710 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T12:17:43.002 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-21T12:17:43.003 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi079.front.sepia.ceph.com' 2024-09-21T12:17:43.003 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi079.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T12:17:53.135 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.104 2024-09-21T12:18:01.326 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.79 2024-09-21T12:18:02.137 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi104.front.sepia.ceph.com' 2024-09-21T12:18:02.137 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T12:18:05.203 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.104 2024-09-21T12:18:10.330 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi079.front.sepia.ceph.com' 2024-09-21T12:18:10.331 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi079.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T12:18:13.391 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.79 2024-09-21T12:18:17.206 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi104.front.sepia.ceph.com' 2024-09-21T12:18:17.207 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T12:18:25.398 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi079.front.sepia.ceph.com' 2024-09-21T12:18:25.399 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi079.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T12:18:49.045 DEBUG:teuthology.orchestra.run.smithi104:> true 2024-09-21T12:18:49.738 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi104.front.sepia.ceph.com' 2024-09-21T12:18:49.798 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-508.el9.x86_64"... 2024-09-21T12:18:49.799 DEBUG:teuthology.orchestra.run.smithi104:> uname -r 2024-09-21T12:18:49.839 INFO:teuthology.orchestra.run.smithi104.stdout:5.14.0-508.el9.x86_64 2024-09-21T12:18:49.839 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-508.el9.x86_64 vs 5.14.0-508.el9.x86_64 2024-09-21T12:18:49.839 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-21T12:18:49.839 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-21T12:18:50.841 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-09-21T12:18:50.841 DEBUG:teuthology.orchestra.run.smithi104:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-21T12:18:51.117 INFO:teuthology.orchestra.run.smithi104.stdout:ttyS1 2024-09-21T12:18:51.147 DEBUG:teuthology.parallel:result is None 2024-09-21T12:18:57.377 DEBUG:teuthology.orchestra.run.smithi079:> true 2024-09-21T12:18:57.802 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi079.front.sepia.ceph.com' 2024-09-21T12:18:57.802 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-508.el9.x86_64"... 2024-09-21T12:18:57.802 DEBUG:teuthology.orchestra.run.smithi079:> uname -r 2024-09-21T12:18:57.817 INFO:teuthology.orchestra.run.smithi079.stdout:5.14.0-508.el9.x86_64 2024-09-21T12:18:57.817 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-508.el9.x86_64 vs 5.14.0-508.el9.x86_64 2024-09-21T12:18:57.817 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-21T12:18:57.817 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-21T12:18:58.818 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-21T12:18:58.818 DEBUG:teuthology.orchestra.run.smithi079:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-21T12:18:58.936 INFO:teuthology.orchestra.run.smithi079.stdout:ttyS1 2024-09-21T12:18:58.961 DEBUG:teuthology.parallel:result is None 2024-09-21T12:18:58.961 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-21T12:18:58.968 INFO:teuthology.task.internal:Creating test directory... 2024-09-21T12:18:58.968 DEBUG:teuthology.orchestra.run.smithi079:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-21T12:18:58.970 DEBUG:teuthology.orchestra.run.smithi104:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-21T12:18:58.989 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-21T12:18:59.002 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-21T12:18:59.029 INFO:teuthology.task.internal:Creating archive directory... 2024-09-21T12:18:59.029 DEBUG:teuthology.orchestra.run.smithi079:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-21T12:18:59.031 DEBUG:teuthology.orchestra.run.smithi104:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-21T12:18:59.077 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-21T12:18:59.084 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-21T12:18:59.084 DEBUG:teuthology.orchestra.run.smithi079:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-21T12:18:59.089 DEBUG:teuthology.orchestra.run.smithi104:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-21T12:18:59.113 INFO:teuthology.orchestra.run.smithi079.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-21T12:18:59.125 INFO:teuthology.orchestra.run.smithi079.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-21T12:18:59.175 INFO:teuthology.orchestra.run.smithi104.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-21T12:18:59.186 INFO:teuthology.orchestra.run.smithi104.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-21T12:18:59.188 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-21T12:18:59.196 INFO:teuthology.task.internal:Configuring sudo... 2024-09-21T12:18:59.196 DEBUG:teuthology.orchestra.run.smithi079:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-21T12:18:59.198 DEBUG:teuthology.orchestra.run.smithi104:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-21T12:18:59.225 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-21T12:18:59.234 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-21T12:18:59.234 DEBUG:teuthology.orchestra.run.smithi079:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-21T12:18:59.260 DEBUG:teuthology.orchestra.run.smithi104:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-21T12:18:59.282 DEBUG:teuthology.orchestra.run.smithi079:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-21T12:18:59.350 DEBUG:teuthology.orchestra.run.smithi079:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-21T12:18:59.398 DEBUG:teuthology.orchestra.run.smithi079:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-21T12:18:59.478 DEBUG:teuthology.orchestra.run.smithi079:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-21T12:18:59.513 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-09-21T12:18:59.513 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-21T12:18:59.574 DEBUG:teuthology.orchestra.run.smithi104:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-21T12:18:59.620 DEBUG:teuthology.orchestra.run.smithi104:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-21T12:18:59.671 DEBUG:teuthology.orchestra.run.smithi104:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-21T12:18:59.751 DEBUG:teuthology.orchestra.run.smithi104:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-21T12:18:59.799 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-09-21T12:18:59.799 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-21T12:18:59.866 DEBUG:teuthology.orchestra.run.smithi079:> sudo service rsyslog restart 2024-09-21T12:18:59.868 DEBUG:teuthology.orchestra.run.smithi104:> sudo service rsyslog restart 2024-09-21T12:18:59.935 INFO:teuthology.orchestra.run.smithi079.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-21T12:18:59.955 INFO:teuthology.orchestra.run.smithi104.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-21T12:19:00.469 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-21T12:19:00.476 INFO:teuthology.task.internal:Starting timer... 2024-09-21T12:19:00.476 INFO:teuthology.run_tasks:Running task pcp... 2024-09-21T12:19:00.486 INFO:teuthology.run_tasks:Running task selinux... 2024-09-21T12:19:00.494 DEBUG:teuthology.orchestra.run.smithi079:> sudo service auditd rotate 2024-09-21T12:19:00.613 INFO:teuthology.orchestra.run.smithi079.stdout:Rotating logs: 2024-09-21T12:19:00.618 DEBUG:teuthology.orchestra.run.smithi104:> sudo service auditd rotate 2024-09-21T12:19:00.678 INFO:teuthology.orchestra.run.smithi104.stdout:Rotating logs: 2024-09-21T12:19:00.680 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-21T12:19:00.680 DEBUG:teuthology.orchestra.run.smithi079:> /usr/sbin/getenforce 2024-09-21T12:19:00.752 INFO:teuthology.orchestra.run.smithi079.stdout:Permissive 2024-09-21T12:19:00.753 DEBUG:teuthology.orchestra.run.smithi104:> /usr/sbin/getenforce 2024-09-21T12:19:00.785 INFO:teuthology.orchestra.run.smithi104.stdout:Permissive 2024-09-21T12:19:00.785 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi079.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi104.front.sepia.ceph.com': 'permissive'} 2024-09-21T12:19:00.785 DEBUG:teuthology.orchestra.run.smithi079:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-21T12:19:00.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:19:00.830 DEBUG:teuthology.orchestra.run.smithi104:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-21T12:19:00.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:19:00.859 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-21T12:19:00.859 DEBUG:teuthology.orchestra.run.smithi079:> sudo /usr/sbin/setenforce permissive 2024-09-21T12:19:01.062 DEBUG:teuthology.orchestra.run.smithi104:> sudo /usr/sbin/setenforce permissive 2024-09-21T12:19:01.098 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-21T12:19:01.113 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-09-21T12:19:01.116 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-09-21T12:19:01.229 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-21T12:19:01.238 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-21T12:19:01.239 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi079.front.sepia.ceph.com,smithi104.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-21T12:22:41.741 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi079.front.sepia.ceph.com'), Remote(name='ubuntu@smithi104.front.sepia.ceph.com')] 2024-09-21T12:22:41.742 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi079.front.sepia.ceph.com' 2024-09-21T12:22:41.742 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi079.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T12:22:41.816 DEBUG:teuthology.orchestra.run.smithi079:> true 2024-09-21T12:22:41.906 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi079.front.sepia.ceph.com' 2024-09-21T12:22:41.906 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi104.front.sepia.ceph.com' 2024-09-21T12:22:41.907 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi104.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T12:22:41.982 DEBUG:teuthology.orchestra.run.smithi104:> true 2024-09-21T12:22:42.058 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi104.front.sepia.ceph.com' 2024-09-21T12:22:42.058 INFO:teuthology.run_tasks:Running task clock... 2024-09-21T12:22:42.068 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-21T12:22:42.068 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-21T12:22:42.068 DEBUG:teuthology.orchestra.run.smithi079:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-21T12:22:42.071 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-21T12:22:42.071 DEBUG:teuthology.orchestra.run.smithi104:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-21T12:22:42.098 INFO:teuthology.orchestra.run.smithi079.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-21T12:22:42.113 INFO:teuthology.orchestra.run.smithi079.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-21T12:22:42.128 INFO:teuthology.orchestra.run.smithi104.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-21T12:22:42.137 INFO:teuthology.orchestra.run.smithi079.stderr:sudo: ntpd: command not found 2024-09-21T12:22:42.142 INFO:teuthology.orchestra.run.smithi104.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-21T12:22:42.148 INFO:teuthology.orchestra.run.smithi079.stdout:506 Cannot talk to daemon 2024-09-21T12:22:42.163 INFO:teuthology.orchestra.run.smithi079.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-21T12:22:42.166 INFO:teuthology.orchestra.run.smithi104.stderr:sudo: ntpd: command not found 2024-09-21T12:22:42.177 INFO:teuthology.orchestra.run.smithi079.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-21T12:22:42.177 INFO:teuthology.orchestra.run.smithi104.stdout:506 Cannot talk to daemon 2024-09-21T12:22:42.191 INFO:teuthology.orchestra.run.smithi104.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-21T12:22:42.205 INFO:teuthology.orchestra.run.smithi104.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-21T12:22:42.240 INFO:teuthology.orchestra.run.smithi079.stderr:bash: line 1: ntpq: command not found 2024-09-21T12:22:42.243 INFO:teuthology.orchestra.run.smithi079.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-21T12:22:42.243 INFO:teuthology.orchestra.run.smithi079.stdout:=============================================================================== 2024-09-21T12:22:42.243 INFO:teuthology.orchestra.run.smithi079.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T12:22:42.243 INFO:teuthology.orchestra.run.smithi079.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T12:22:42.243 INFO:teuthology.orchestra.run.smithi079.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T12:22:42.244 INFO:teuthology.orchestra.run.smithi079.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T12:22:42.260 INFO:teuthology.orchestra.run.smithi104.stderr:bash: line 1: ntpq: command not found 2024-09-21T12:22:42.265 INFO:teuthology.orchestra.run.smithi104.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-21T12:22:42.265 INFO:teuthology.orchestra.run.smithi104.stdout:=============================================================================== 2024-09-21T12:22:42.265 INFO:teuthology.orchestra.run.smithi104.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T12:22:42.265 INFO:teuthology.orchestra.run.smithi104.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T12:22:42.265 INFO:teuthology.orchestra.run.smithi104.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T12:22:42.265 INFO:teuthology.orchestra.run.smithi104.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T12:22:42.265 INFO:teuthology.run_tasks:Running task cephadm... 2024-09-21T12:22:42.356 INFO:tasks.cephadm:Config: {'compiled_cephadm_branch': 'reef', 'image': 'quay.ceph.io/ceph-ci/ceph:reef', 'conf': {'global': {'mon election default strategy': 1}, 'mgr': {'debug mgr': 20, 'debug ms': 1, 'mgr/cephadm/use_agent': True}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'CEPHADM_STRAY_DAEMON', 'CEPHADM_FAILED_DAEMON', 'CEPHADM_AGENT_DOWN'], 'log-only-match': ['CEPHADM_'], 'sha1': '441784946648ff53dd310293f9dd752afeeea72a'} 2024-09-21T12:22:42.356 INFO:tasks.cephadm:Cluster image is quay.ceph.io/ceph-ci/ceph:reef 2024-09-21T12:22:42.356 INFO:tasks.cephadm:Cluster fsid is 33168022-7814-11ef-baf6-efdc52797490 2024-09-21T12:22:42.356 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-09-21T12:22:42.357 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.79', 'mon.c': '[v2:172.21.15.79:3301,v1:172.21.15.79:6790]', 'mon.b': '172.21.15.104'} 2024-09-21T12:22:42.357 INFO:tasks.cephadm:First mon is mon.a on smithi079 2024-09-21T12:22:42.357 INFO:tasks.cephadm:First mgr is y 2024-09-21T12:22:42.357 INFO:tasks.cephadm:Normalizing hostnames... 2024-09-21T12:22:42.357 DEBUG:teuthology.orchestra.run.smithi079:> sudo hostname $(hostname -s) 2024-09-21T12:22:42.384 DEBUG:teuthology.orchestra.run.smithi104:> sudo hostname $(hostname -s) 2024-09-21T12:22:42.409 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra for reef 2024-09-21T12:22:42.409 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=441784946648ff53dd310293f9dd752afeeea72a 2024-09-21T12:22:42.555 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '441784946648ff53dd310293f9dd752afeeea72a', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/83203/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.3.0-5104-g44178494', 'node_name': '172.21.5.36+adami06', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.3.0-5104.g44178494'}, 'url': 'https://3.chacra.ceph.com/r/ceph/wip-guits-main-2024-09-21-0842/441784946648ff53dd310293f9dd752afeeea72a/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-09-21 10:26:40.656703', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-guits-main-2024-09-21-0842', 'chacra_url': 'https://3.chacra.ceph.com/repos/ceph/wip-guits-main-2024-09-21-0842/441784946648ff53dd310293f9dd752afeeea72a/centos/9/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2024-09-21T12:22:42.832 INFO:tasks.util.chacra:got chacra host 4.chacra.ceph.com, ref reef, sha1 6a1bdf854bb5803efd58957907442aa01f41a8fc from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&ref=reef 2024-09-21T12:22:42.833 INFO:tasks.cephadm:Discovered cachra url: https://4.chacra.ceph.com/binaries/ceph/reef/6a1bdf854bb5803efd58957907442aa01f41a8fc/centos/9/x86_64/flavors/default/cephadm 2024-09-21T12:22:42.833 INFO:tasks.cephadm:Downloading cephadm from url: https://4.chacra.ceph.com/binaries/ceph/reef/6a1bdf854bb5803efd58957907442aa01f41a8fc/centos/9/x86_64/flavors/default/cephadm 2024-09-21T12:22:42.833 DEBUG:teuthology.orchestra.run.smithi079:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/reef/6a1bdf854bb5803efd58957907442aa01f41a8fc/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-09-21T12:22:43.317 INFO:teuthology.orchestra.run.smithi079.stdout:-rw-r--r--. 1 ubuntu ubuntu 217222 Sep 21 12:22 /home/ubuntu/cephtest/cephadm 2024-09-21T12:22:43.317 DEBUG:teuthology.orchestra.run.smithi104:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/reef/6a1bdf854bb5803efd58957907442aa01f41a8fc/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-09-21T12:22:43.793 INFO:teuthology.orchestra.run.smithi104.stdout:-rw-r--r--. 1 ubuntu ubuntu 217222 Sep 21 12:22 /home/ubuntu/cephtest/cephadm 2024-09-21T12:22:43.794 DEBUG:teuthology.orchestra.run.smithi079:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-09-21T12:22:43.813 DEBUG:teuthology.orchestra.run.smithi104:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-09-21T12:22:43.842 INFO:tasks.cephadm:Pulling image quay.ceph.io/ceph-ci/ceph:reef on all hosts... 2024-09-21T12:22:43.843 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef pull 2024-09-21T12:22:43.856 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef pull 2024-09-21T12:22:45.400 INFO:teuthology.orchestra.run.smithi104.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:reef... 2024-09-21T12:22:45.708 INFO:teuthology.orchestra.run.smithi079.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:reef... 2024-09-21T12:23:11.980 INFO:teuthology.orchestra.run.smithi104.stdout:{ 2024-09-21T12:23:11.980 INFO:teuthology.orchestra.run.smithi104.stdout: "ceph_version": "ceph version 18.2.4-871-g6a1bdf85 (6a1bdf854bb5803efd58957907442aa01f41a8fc) reef (stable)", 2024-09-21T12:23:11.980 INFO:teuthology.orchestra.run.smithi104.stdout: "image_id": "0adab276c97e6f3143ffbca5eeff37fcd7dc3b5c0f339a9f747381d9993602de", 2024-09-21T12:23:11.980 INFO:teuthology.orchestra.run.smithi104.stdout: "repo_digests": [ 2024-09-21T12:23:11.980 INFO:teuthology.orchestra.run.smithi104.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6" 2024-09-21T12:23:11.980 INFO:teuthology.orchestra.run.smithi104.stdout: ] 2024-09-21T12:23:11.980 INFO:teuthology.orchestra.run.smithi104.stdout:} 2024-09-21T12:23:13.154 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-21T12:23:13.155 INFO:teuthology.orchestra.run.smithi079.stdout: "ceph_version": "ceph version 18.2.4-871-g6a1bdf85 (6a1bdf854bb5803efd58957907442aa01f41a8fc) reef (stable)", 2024-09-21T12:23:13.155 INFO:teuthology.orchestra.run.smithi079.stdout: "image_id": "0adab276c97e6f3143ffbca5eeff37fcd7dc3b5c0f339a9f747381d9993602de", 2024-09-21T12:23:13.155 INFO:teuthology.orchestra.run.smithi079.stdout: "repo_digests": [ 2024-09-21T12:23:13.155 INFO:teuthology.orchestra.run.smithi079.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6" 2024-09-21T12:23:13.155 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-21T12:23:13.155 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-21T12:23:13.173 DEBUG:teuthology.orchestra.run.smithi079:> sudo mkdir -p /etc/ceph 2024-09-21T12:23:13.214 DEBUG:teuthology.orchestra.run.smithi104:> sudo mkdir -p /etc/ceph 2024-09-21T12:23:13.244 DEBUG:teuthology.orchestra.run.smithi079:> sudo chmod 777 /etc/ceph 2024-09-21T12:23:13.278 DEBUG:teuthology.orchestra.run.smithi104:> sudo chmod 777 /etc/ceph 2024-09-21T12:23:13.310 INFO:tasks.cephadm:Writing seed config... 2024-09-21T12:23:13.311 INFO:tasks.cephadm: override: [global] mon election default strategy = 1 2024-09-21T12:23:13.312 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-09-21T12:23:13.312 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-09-21T12:23:13.312 INFO:tasks.cephadm: override: [mgr] mgr/cephadm/use_agent = True 2024-09-21T12:23:13.312 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-09-21T12:23:13.312 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-09-21T12:23:13.312 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-09-21T12:23:13.312 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-09-21T12:23:13.312 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-09-21T12:23:13.312 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-09-21T12:23:13.313 DEBUG:teuthology.orchestra.run.smithi079:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-09-21T12:23:13.332 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 33168022-7814-11ef-baf6-efdc52797490 mon election default strategy = 1 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 mgr/cephadm/use_agent = True [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true 2024-09-21T12:23:13.333 DEBUG:teuthology.orchestra.run.smithi079:mon.a> sudo journalctl -f -n 0 -u ceph-33168022-7814-11ef-baf6-efdc52797490@mon.a.service 2024-09-21T12:23:13.375 DEBUG:teuthology.orchestra.run.smithi079:mgr.y> sudo journalctl -f -n 0 -u ceph-33168022-7814-11ef-baf6-efdc52797490@mgr.y.service 2024-09-21T12:23:13.417 INFO:tasks.cephadm:Bootstrapping... 2024-09-21T12:23:13.417 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef -v bootstrap --fsid 33168022-7814-11ef-baf6-efdc52797490 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.79 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-09-21T12:23:13.569 INFO:teuthology.orchestra.run.smithi079.stdout:-------------------------------------------------------------------------------- 2024-09-21T12:23:13.569 INFO:teuthology.orchestra.run.smithi079.stdout:cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:reef', '-v', 'bootstrap', '--fsid', '33168022-7814-11ef-baf6-efdc52797490', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.79', '--skip-admin-label'] 2024-09-21T12:23:13.592 INFO:teuthology.orchestra.run.smithi079.stdout:/bin/podman: stdout 5.2.2 2024-09-21T12:23:13.592 INFO:teuthology.orchestra.run.smithi079.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-09-21T12:23:13.593 INFO:teuthology.orchestra.run.smithi079.stdout:Verifying podman|docker is present... 2024-09-21T12:23:13.614 INFO:teuthology.orchestra.run.smithi079.stdout:/bin/podman: stdout 5.2.2 2024-09-21T12:23:13.615 INFO:teuthology.orchestra.run.smithi079.stdout:Verifying lvm2 is present... 2024-09-21T12:23:13.615 INFO:teuthology.orchestra.run.smithi079.stdout:Verifying time synchronization is in place... 2024-09-21T12:23:13.622 INFO:teuthology.orchestra.run.smithi079.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-09-21T12:23:13.622 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-09-21T12:23:13.628 INFO:teuthology.orchestra.run.smithi079.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-09-21T12:23:13.628 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stdout inactive 2024-09-21T12:23:13.634 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stdout enabled 2024-09-21T12:23:13.640 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stdout active 2024-09-21T12:23:13.641 INFO:teuthology.orchestra.run.smithi079.stdout:Unit chronyd.service is enabled and running 2024-09-21T12:23:13.641 INFO:teuthology.orchestra.run.smithi079.stdout:Repeating the final host check... 2024-09-21T12:23:13.662 INFO:teuthology.orchestra.run.smithi079.stdout:/bin/podman: stdout 5.2.2 2024-09-21T12:23:13.662 INFO:teuthology.orchestra.run.smithi079.stdout:podman (/bin/podman) version 5.2.2 is present 2024-09-21T12:23:13.662 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl is present 2024-09-21T12:23:13.662 INFO:teuthology.orchestra.run.smithi079.stdout:lvcreate is present 2024-09-21T12:23:13.668 INFO:teuthology.orchestra.run.smithi079.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-09-21T12:23:13.668 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-09-21T12:23:13.674 INFO:teuthology.orchestra.run.smithi079.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-09-21T12:23:13.674 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stdout inactive 2024-09-21T12:23:13.680 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stdout enabled 2024-09-21T12:23:13.687 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stdout active 2024-09-21T12:23:13.687 INFO:teuthology.orchestra.run.smithi079.stdout:Unit chronyd.service is enabled and running 2024-09-21T12:23:13.687 INFO:teuthology.orchestra.run.smithi079.stdout:Host looks OK 2024-09-21T12:23:13.687 INFO:teuthology.orchestra.run.smithi079.stdout:Cluster fsid: 33168022-7814-11ef-baf6-efdc52797490 2024-09-21T12:23:13.687 INFO:teuthology.orchestra.run.smithi079.stdout:Acquiring lock 140656611815584 on /run/cephadm/33168022-7814-11ef-baf6-efdc52797490.lock 2024-09-21T12:23:13.687 INFO:teuthology.orchestra.run.smithi079.stdout:Lock 140656611815584 acquired on /run/cephadm/33168022-7814-11ef-baf6-efdc52797490.lock 2024-09-21T12:23:13.688 INFO:teuthology.orchestra.run.smithi079.stdout:Verifying IP 172.21.15.79 port 3300 ... 2024-09-21T12:23:13.688 INFO:teuthology.orchestra.run.smithi079.stdout:Verifying IP 172.21.15.79 port 6789 ... 2024-09-21T12:23:13.688 INFO:teuthology.orchestra.run.smithi079.stdout:Base mon IP(s) is [172.21.15.79:3300, 172.21.15.79:6789], mon addrv is [v2:172.21.15.79:3300,v1:172.21.15.79:6789] 2024-09-21T12:23:13.691 INFO:teuthology.orchestra.run.smithi079.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.79 metric 100 2024-09-21T12:23:13.691 INFO:teuthology.orchestra.run.smithi079.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.79 metric 100 2024-09-21T12:23:13.694 INFO:teuthology.orchestra.run.smithi079.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-09-21T12:23:13.694 INFO:teuthology.orchestra.run.smithi079.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 1024 pref medium 2024-09-21T12:23:13.694 INFO:teuthology.orchestra.run.smithi079.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 100 pref medium 2024-09-21T12:23:13.696 INFO:teuthology.orchestra.run.smithi079.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-09-21T12:23:13.696 INFO:teuthology.orchestra.run.smithi079.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-09-21T12:23:13.696 INFO:teuthology.orchestra.run.smithi079.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-09-21T12:23:13.696 INFO:teuthology.orchestra.run.smithi079.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-09-21T12:23:13.696 INFO:teuthology.orchestra.run.smithi079.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe8f:ce87/64 scope link noprefixroute 2024-09-21T12:23:13.697 INFO:teuthology.orchestra.run.smithi079.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-09-21T12:23:13.697 INFO:teuthology.orchestra.run.smithi079.stdout:Mon IP `172.21.15.79` is in CIDR network `172.21.0.0/20` 2024-09-21T12:23:13.697 INFO:teuthology.orchestra.run.smithi079.stdout:Mon IP `172.21.15.79` is in CIDR network `172.21.0.0/20` 2024-09-21T12:23:13.697 INFO:teuthology.orchestra.run.smithi079.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-09-21T12:23:13.697 INFO:teuthology.orchestra.run.smithi079.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-09-21T12:23:13.697 INFO:teuthology.orchestra.run.smithi079.stdout:Pulling container image quay.ceph.io/ceph-ci/ceph:reef... 2024-09-21T12:23:14.165 INFO:teuthology.orchestra.run.smithi079.stdout:/bin/podman: stdout 0adab276c97e6f3143ffbca5eeff37fcd7dc3b5c0f339a9f747381d9993602de 2024-09-21T12:23:14.166 INFO:teuthology.orchestra.run.smithi079.stdout:/bin/podman: stderr Trying to pull quay.ceph.io/ceph-ci/ceph:reef... 2024-09-21T12:23:14.166 INFO:teuthology.orchestra.run.smithi079.stdout:/bin/podman: stderr Getting image source signatures 2024-09-21T12:23:14.166 INFO:teuthology.orchestra.run.smithi079.stdout:/bin/podman: stderr Copying blob sha256:b51f6d1e10fe8de9e06c08c0a23ce0010d2bef42d30a6f89f3ac07fd857715ba 2024-09-21T12:23:14.166 INFO:teuthology.orchestra.run.smithi079.stdout:/bin/podman: stderr Copying blob sha256:e8b54c863393b7a8216a71737771b73b16a8e43ec7acf927c7faa175c255e551 2024-09-21T12:23:14.166 INFO:teuthology.orchestra.run.smithi079.stdout:/bin/podman: stderr Copying config sha256:0adab276c97e6f3143ffbca5eeff37fcd7dc3b5c0f339a9f747381d9993602de 2024-09-21T12:23:14.166 INFO:teuthology.orchestra.run.smithi079.stdout:/bin/podman: stderr Writing manifest to image destination 2024-09-21T12:23:15.070 INFO:teuthology.orchestra.run.smithi079.stdout:ceph: stdout ceph version 18.2.4-871-g6a1bdf85 (6a1bdf854bb5803efd58957907442aa01f41a8fc) reef (stable) 2024-09-21T12:23:15.071 INFO:teuthology.orchestra.run.smithi079.stdout:Ceph version: ceph version 18.2.4-871-g6a1bdf85 (6a1bdf854bb5803efd58957907442aa01f41a8fc) reef (stable) 2024-09-21T12:23:15.071 INFO:teuthology.orchestra.run.smithi079.stdout:Extracting ceph user uid/gid from container image... 2024-09-21T12:23:16.061 INFO:teuthology.orchestra.run.smithi079.stdout:stat: stdout 167 167 2024-09-21T12:23:16.061 INFO:teuthology.orchestra.run.smithi079.stdout:Creating initial keys... 2024-09-21T12:23:16.958 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph-authtool: stdout AQC0uu5ml4/YGBAAR3rO10UdhjE0vp3uxnTTYQ== 2024-09-21T12:23:17.832 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph-authtool: stdout AQC1uu5mAt/SDxAAhyMzF1jyR/B4vUcwKlNdxw== 2024-09-21T12:23:18.821 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph-authtool: stdout AQC2uu5mRv44CBAAF54I0EoGkTUYBSCekjQuog== 2024-09-21T12:23:18.822 INFO:teuthology.orchestra.run.smithi079.stdout:Creating initial monmap... 2024-09-21T12:23:19.644 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-09-21T12:23:19.645 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = pacific 2024-09-21T12:23:19.645 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 33168022-7814-11ef-baf6-efdc52797490 2024-09-21T12:23:19.645 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-09-21T12:23:19.645 INFO:teuthology.orchestra.run.smithi079.stdout:monmaptool for a [v2:172.21.15.79:3300,v1:172.21.15.79:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-09-21T12:23:19.645 INFO:teuthology.orchestra.run.smithi079.stdout:setting min_mon_release = pacific 2024-09-21T12:23:19.645 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/monmaptool: set fsid to 33168022-7814-11ef-baf6-efdc52797490 2024-09-21T12:23:19.645 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-09-21T12:23:19.645 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-21T12:23:19.646 INFO:teuthology.orchestra.run.smithi079.stdout:Creating mon... 2024-09-21T12:23:20.576 INFO:teuthology.orchestra.run.smithi079.stdout:create mon.a on 2024-09-21T12:23:20.930 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-09-21T12:23:21.102 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-33168022-7814-11ef-baf6-efdc52797490.target → /etc/systemd/system/ceph-33168022-7814-11ef-baf6-efdc52797490.target. 2024-09-21T12:23:21.102 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-33168022-7814-11ef-baf6-efdc52797490.target → /etc/systemd/system/ceph-33168022-7814-11ef-baf6-efdc52797490.target. 2024-09-21T12:23:21.315 INFO:teuthology.orchestra.run.smithi079.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-33168022-7814-11ef-baf6-efdc52797490@mon.a 2024-09-21T12:23:21.315 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stderr Failed to reset failed state of unit ceph-33168022-7814-11ef-baf6-efdc52797490@mon.a.service: Unit ceph-33168022-7814-11ef-baf6-efdc52797490@mon.a.service not loaded. 2024-09-21T12:23:21.467 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-33168022-7814-11ef-baf6-efdc52797490.target.wants/ceph-33168022-7814-11ef-baf6-efdc52797490@mon.a.service → /etc/systemd/system/ceph-33168022-7814-11ef-baf6-efdc52797490@.service. 2024-09-21T12:23:21.957 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:21 smithi079 ceph-mon[23320]: mon.a is new leader, mons a in quorum (ranks 0) 2024-09-21T12:23:21.957 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:21 smithi079 bash[23302]: 9f8c062e7edd8b351115ac8bdd241f2d9b0dd0be87b24689d3dff83acad6bd43 2024-09-21T12:23:21.969 INFO:teuthology.orchestra.run.smithi079.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-21T12:23:21.969 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stdout disabled 2024-09-21T12:23:21.976 INFO:teuthology.orchestra.run.smithi079.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-21T12:23:21.976 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stdout inactive 2024-09-21T12:23:21.976 INFO:teuthology.orchestra.run.smithi079.stdout:firewalld.service is not enabled 2024-09-21T12:23:21.977 INFO:teuthology.orchestra.run.smithi079.stdout:Not possible to enable service . firewalld.service is not available 2024-09-21T12:23:21.977 INFO:teuthology.orchestra.run.smithi079.stdout:Waiting for mon to start... 2024-09-21T12:23:21.977 INFO:teuthology.orchestra.run.smithi079.stdout:Waiting for mon... 2024-09-21T12:23:22.221 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:21 smithi079 systemd[1]: Started Ceph mon.a for 33168022-7814-11ef-baf6-efdc52797490. 2024-09-21T12:23:22.993 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout cluster: 2024-09-21T12:23:22.993 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout id: 33168022-7814-11ef-baf6-efdc52797490 2024-09-21T12:23:22.993 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-09-21T12:23:22.993 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout 2024-09-21T12:23:22.993 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout services: 2024-09-21T12:23:22.993 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.561301s) 2024-09-21T12:23:22.993 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-09-21T12:23:22.993 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-09-21T12:23:22.993 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout 2024-09-21T12:23:22.993 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout data: 2024-09-21T12:23:22.993 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-09-21T12:23:22.994 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-09-21T12:23:22.994 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-09-21T12:23:22.994 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout pgs: 2024-09-21T12:23:22.994 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout 2024-09-21T12:23:22.994 INFO:teuthology.orchestra.run.smithi079.stdout:mon is available 2024-09-21T12:23:22.994 INFO:teuthology.orchestra.run.smithi079.stdout:Assimilating anything we can from ceph.conf... 2024-09-21T12:23:23.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:22 smithi079 ceph-mon[23320]: mon.a is new leader, mons a in quorum (ranks 0) 2024-09-21T12:23:23.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:22 smithi079 ceph-mon[23320]: monmap e1: 1 mons at {a=[v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-21T12:23:23.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:22 smithi079 ceph-mon[23320]: fsmap 2024-09-21T12:23:23.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:22 smithi079 ceph-mon[23320]: osdmap e1: 0 total, 0 up, 0 in 2024-09-21T12:23:23.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:22 smithi079 ceph-mon[23320]: mgrmap e1: no daemons active 2024-09-21T12:23:23.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:22 smithi079 ceph-mon[23320]: from='client.? 172.21.15.79:0/2640291404' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-09-21T12:23:23.957 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout 2024-09-21T12:23:23.957 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout [global] 2024-09-21T12:23:23.957 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout fsid = 33168022-7814-11ef-baf6-efdc52797490 2024-09-21T12:23:23.957 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.79:3300,v1:172.21.15.79:6789] 2024-09-21T12:23:23.958 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-09-21T12:23:23.958 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-09-21T12:23:23.958 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-09-21T12:23:23.958 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-09-21T12:23:23.958 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout 2024-09-21T12:23:23.958 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout [mgr] 2024-09-21T12:23:23.958 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout mgr/cephadm/use_agent = True 2024-09-21T12:23:23.958 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-09-21T12:23:23.958 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout 2024-09-21T12:23:23.958 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout [osd] 2024-09-21T12:23:23.958 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-09-21T12:23:23.958 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-09-21T12:23:23.958 INFO:teuthology.orchestra.run.smithi079.stdout:Generating new minimal ceph.conf... 2024-09-21T12:23:24.159 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:23 smithi079 ceph-mon[23320]: from='client.? 172.21.15.79:0/540012544' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-09-21T12:23:24.159 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:23 smithi079 ceph-mon[23320]: from='client.? 172.21.15.79:0/540012544' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-09-21T12:23:24.923 INFO:teuthology.orchestra.run.smithi079.stdout:Restarting the monitor... 2024-09-21T12:23:25.189 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:24 smithi079 ceph-mon[23320]: from='client.? 172.21.15.79:0/3471094176' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:23:25.189 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:24 smithi079 systemd[1]: Stopping Ceph mon.a for 33168022-7814-11ef-baf6-efdc52797490... 2024-09-21T12:23:25.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:25 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mon-a[23316]: 2024-09-21T12:23:25.187+0000 7fe7dba23640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T12:23:25.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:25 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mon-a[23316]: 2024-09-21T12:23:25.187+0000 7fe7dba23640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-09-21T12:23:25.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:25 smithi079 podman[23574]: 2024-09-21 12:23:25.412519575 +0000 UTC m=+0.330109006 container died 9f8c062e7edd8b351115ac8bdd241f2d9b0dd0be87b24689d3dff83acad6bd43 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-33168022-7814-11ef-baf6-efdc52797490-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, RELEASE=reef-6a1bdf8, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD) 2024-09-21T12:23:25.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:25 smithi079 podman[23574]: 2024-09-21 12:23:25.762618006 +0000 UTC m=+0.680207479 container cleanup 9f8c062e7edd8b351115ac8bdd241f2d9b0dd0be87b24689d3dff83acad6bd43 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-33168022-7814-11ef-baf6-efdc52797490-mon-a, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, RELEASE=reef-6a1bdf8, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-09-21T12:23:25.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:25 smithi079 bash[23574]: ceph-33168022-7814-11ef-baf6-efdc52797490-mon-a 2024-09-21T12:23:26.438 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 podman[23586]: 2024-09-21 12:23:26.055398599 +0000 UTC m=+0.634497453 container remove 9f8c062e7edd8b351115ac8bdd241f2d9b0dd0be87b24689d3dff83acad6bd43 (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-33168022-7814-11ef-baf6-efdc52797490-mon-a, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, RELEASE=reef-6a1bdf8, ceph=True, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-09-21T12:23:26.439 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 systemd[1]: ceph-33168022-7814-11ef-baf6-efdc52797490@mon.a.service: Deactivated successfully. 2024-09-21T12:23:26.439 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 systemd[1]: Stopped Ceph mon.a for 33168022-7814-11ef-baf6-efdc52797490. 2024-09-21T12:23:26.439 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 systemd[1]: Starting Ceph mon.a for 33168022-7814-11ef-baf6-efdc52797490... 2024-09-21T12:23:26.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 podman[23680]: 2024-09-21 12:23:26.339960495 +0000 UTC m=+0.015755546 image pull 0adab276c97e6f3143ffbca5eeff37fcd7dc3b5c0f339a9f747381d9993602de quay.ceph.io/ceph-ci/ceph:reef 2024-09-21T12:23:26.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 podman[23680]: 2024-09-21 12:23:26.439373767 +0000 UTC m=+0.115168804 container create 794d66ac29c5f080f3d25c707ee3cb754cd838c130700defb586dd0788bdd96a (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-33168022-7814-11ef-baf6-efdc52797490-mon-a, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, RELEASE=reef-6a1bdf8, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T12:23:26.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 podman[23680]: 2024-09-21 12:23:26.650306519 +0000 UTC m=+0.326101557 container init 794d66ac29c5f080f3d25c707ee3cb754cd838c130700defb586dd0788bdd96a (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-33168022-7814-11ef-baf6-efdc52797490-mon-a, ceph=True, RELEASE=reef-6a1bdf8, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731) 2024-09-21T12:23:26.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 podman[23680]: 2024-09-21 12:23:26.653656051 +0000 UTC m=+0.329451091 container start 794d66ac29c5f080f3d25c707ee3cb754cd838c130700defb586dd0788bdd96a (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-33168022-7814-11ef-baf6-efdc52797490-mon-a, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=reef-6a1bdf8, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-09-21T12:23:26.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: set uid:gid to 167:167 (ceph:ceph) 2024-09-21T12:23:26.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: ceph version 18.2.4-871-g6a1bdf85 (6a1bdf854bb5803efd58957907442aa01f41a8fc) reef (stable), process ceph-mon, pid 2 2024-09-21T12:23:26.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: pidfile_write: ignore empty --pid-file 2024-09-21T12:23:26.693 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: load: jerasure load: lrc 2024-09-21T12:23:26.693 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: RocksDB version: 7.9.2 2024-09-21T12:23:26.693 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Git sha 0 2024-09-21T12:23:26.693 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Compile date 2024-09-20 20:34:00 2024-09-21T12:23:26.693 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: DB SUMMARY 2024-09-21T12:23:26.693 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: DB Session ID: TSDRK3JJIYZ5IQIC7ZHC 2024-09-21T12:23:26.693 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: CURRENT file: CURRENT 2024-09-21T12:23:26.693 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: IDENTITY file: IDENTITY 2024-09-21T12:23:26.693 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-09-21T12:23:26.693 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-09-21T12:23:26.693 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 83640 ; 2024-09-21T12:23:26.693 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.error_if_exists: 0 2024-09-21T12:23:26.693 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.create_if_missing: 0 2024-09-21T12:23:26.693 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.paranoid_checks: 1 2024-09-21T12:23:26.694 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.flush_verify_memtable_count: 1 2024-09-21T12:23:26.695 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-21T12:23:26.695 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-09-21T12:23:26.695 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.env: 0x557c9144bbc0 2024-09-21T12:23:26.695 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.fs: PosixFileSystem 2024-09-21T12:23:26.695 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.info_log: 0x557c92670be0 2024-09-21T12:23:26.695 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.max_file_opening_threads: 16 2024-09-21T12:23:26.695 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.statistics: (nil) 2024-09-21T12:23:26.695 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.use_fsync: 0 2024-09-21T12:23:26.695 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.max_log_file_size: 0 2024-09-21T12:23:26.695 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-21T12:23:26.695 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-21T12:23:26.696 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.keep_log_file_num: 1000 2024-09-21T12:23:26.696 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.recycle_log_file_num: 0 2024-09-21T12:23:26.696 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.allow_fallocate: 1 2024-09-21T12:23:26.696 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.allow_mmap_reads: 0 2024-09-21T12:23:26.696 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.allow_mmap_writes: 0 2024-09-21T12:23:26.696 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.use_direct_reads: 0 2024-09-21T12:23:26.696 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-21T12:23:26.696 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.create_missing_column_families: 0 2024-09-21T12:23:26.696 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.db_log_dir: 2024-09-21T12:23:26.696 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.wal_dir: 2024-09-21T12:23:26.696 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-21T12:23:26.696 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-21T12:23:26.696 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-21T12:23:26.696 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-21T12:23:26.696 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-21T12:23:26.697 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-21T12:23:26.697 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.advise_random_on_open: 1 2024-09-21T12:23:26.697 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.db_write_buffer_size: 0 2024-09-21T12:23:26.697 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.write_buffer_manager: 0x557c9267e320 2024-09-21T12:23:26.697 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-21T12:23:26.697 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-21T12:23:26.697 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-21T12:23:26.697 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.rate_limiter: (nil) 2024-09-21T12:23:26.697 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-21T12:23:26.697 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.wal_recovery_mode: 2 2024-09-21T12:23:26.697 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.enable_thread_tracking: 0 2024-09-21T12:23:26.697 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.enable_pipelined_write: 0 2024-09-21T12:23:26.697 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.unordered_write: 0 2024-09-21T12:23:26.697 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-21T12:23:26.697 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-21T12:23:26.698 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-21T12:23:26.698 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-21T12:23:26.698 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.row_cache: None 2024-09-21T12:23:26.698 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.wal_filter: None 2024-09-21T12:23:26.698 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-21T12:23:26.698 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.allow_ingest_behind: 0 2024-09-21T12:23:26.698 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.two_write_queues: 0 2024-09-21T12:23:26.698 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.manual_wal_flush: 0 2024-09-21T12:23:26.698 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.wal_compression: 0 2024-09-21T12:23:26.698 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.atomic_flush: 0 2024-09-21T12:23:26.698 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-21T12:23:26.698 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-21T12:23:26.698 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-21T12:23:26.698 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.log_readahead_size: 0 2024-09-21T12:23:26.699 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-21T12:23:26.699 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.best_efforts_recovery: 0 2024-09-21T12:23:26.699 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-21T12:23:26.699 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-21T12:23:26.699 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.allow_data_in_errors: 0 2024-09-21T12:23:26.699 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.db_host_id: __hostname__ 2024-09-21T12:23:26.699 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.enforce_single_del_contracts: true 2024-09-21T12:23:26.699 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.max_background_jobs: 2 2024-09-21T12:23:26.699 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.max_background_compactions: -1 2024-09-21T12:23:26.699 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.max_subcompactions: 1 2024-09-21T12:23:26.699 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-21T12:23:26.699 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-21T12:23:26.699 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-21T12:23:26.699 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.max_total_wal_size: 0 2024-09-21T12:23:26.699 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-21T12:23:26.700 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-21T12:23:26.700 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-21T12:23:26.700 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-21T12:23:26.700 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.max_open_files: -1 2024-09-21T12:23:26.700 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.bytes_per_sync: 0 2024-09-21T12:23:26.700 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-21T12:23:26.700 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-21T12:23:26.700 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.compaction_readahead_size: 0 2024-09-21T12:23:26.700 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.max_background_flushes: -1 2024-09-21T12:23:26.700 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Compression algorithms supported: 2024-09-21T12:23:26.700 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: kZSTD supported: 0 2024-09-21T12:23:26.700 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: kXpressCompression supported: 0 2024-09-21T12:23:26.700 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: kBZip2Compression supported: 0 2024-09-21T12:23:26.700 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-21T12:23:26.701 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: kLZ4Compression supported: 1 2024-09-21T12:23:26.701 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: kZlibCompression supported: 1 2024-09-21T12:23:26.701 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: kLZ4HCCompression supported: 1 2024-09-21T12:23:26.701 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: kSnappyCompression supported: 1 2024-09-21T12:23:26.701 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-21T12:23:26.701 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: DMutex implementation: pthread_mutex_t 2024-09-21T12:23:26.701 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-09-21T12:23:26.701 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-09-21T12:23:26.701 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-21T12:23:26.701 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.merge_operator: 2024-09-21T12:23:26.701 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.compaction_filter: None 2024-09-21T12:23:26.701 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.compaction_filter_factory: None 2024-09-21T12:23:26.701 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.sst_partitioner_factory: None 2024-09-21T12:23:26.701 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-21T12:23:26.701 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-21T12:23:26.702 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x557c926707e0) 2024-09-21T12:23:26.702 INFO:journalctl@ceph.mon.a.smithi079.stdout: cache_index_and_filter_blocks: 1 2024-09-21T12:23:26.702 INFO:journalctl@ceph.mon.a.smithi079.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-21T12:23:26.702 INFO:journalctl@ceph.mon.a.smithi079.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-21T12:23:26.702 INFO:journalctl@ceph.mon.a.smithi079.stdout: pin_top_level_index_and_filter: 1 2024-09-21T12:23:26.702 INFO:journalctl@ceph.mon.a.smithi079.stdout: index_type: 0 2024-09-21T12:23:26.702 INFO:journalctl@ceph.mon.a.smithi079.stdout: data_block_index_type: 0 2024-09-21T12:23:26.702 INFO:journalctl@ceph.mon.a.smithi079.stdout: index_shortening: 1 2024-09-21T12:23:26.702 INFO:journalctl@ceph.mon.a.smithi079.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-21T12:23:26.702 INFO:journalctl@ceph.mon.a.smithi079.stdout: checksum: 4 2024-09-21T12:23:26.702 INFO:journalctl@ceph.mon.a.smithi079.stdout: no_block_cache: 0 2024-09-21T12:23:26.702 INFO:journalctl@ceph.mon.a.smithi079.stdout: block_cache: 0x557c92665350 2024-09-21T12:23:26.702 INFO:journalctl@ceph.mon.a.smithi079.stdout: block_cache_name: BinnedLRUCache 2024-09-21T12:23:26.702 INFO:journalctl@ceph.mon.a.smithi079.stdout: block_cache_options: 2024-09-21T12:23:26.703 INFO:journalctl@ceph.mon.a.smithi079.stdout: capacity : 536870912 2024-09-21T12:23:26.703 INFO:journalctl@ceph.mon.a.smithi079.stdout: num_shard_bits : 4 2024-09-21T12:23:26.703 INFO:journalctl@ceph.mon.a.smithi079.stdout: strict_capacity_limit : 0 2024-09-21T12:23:26.703 INFO:journalctl@ceph.mon.a.smithi079.stdout: high_pri_pool_ratio: 0.000 2024-09-21T12:23:26.703 INFO:journalctl@ceph.mon.a.smithi079.stdout: block_cache_compressed: (nil) 2024-09-21T12:23:26.703 INFO:journalctl@ceph.mon.a.smithi079.stdout: persistent_cache: (nil) 2024-09-21T12:23:26.703 INFO:journalctl@ceph.mon.a.smithi079.stdout: block_size: 4096 2024-09-21T12:23:26.703 INFO:journalctl@ceph.mon.a.smithi079.stdout: block_size_deviation: 10 2024-09-21T12:23:26.703 INFO:journalctl@ceph.mon.a.smithi079.stdout: block_restart_interval: 16 2024-09-21T12:23:26.703 INFO:journalctl@ceph.mon.a.smithi079.stdout: index_block_restart_interval: 1 2024-09-21T12:23:26.703 INFO:journalctl@ceph.mon.a.smithi079.stdout: metadata_block_size: 4096 2024-09-21T12:23:26.703 INFO:journalctl@ceph.mon.a.smithi079.stdout: partition_filters: 0 2024-09-21T12:23:26.703 INFO:journalctl@ceph.mon.a.smithi079.stdout: use_delta_encoding: 1 2024-09-21T12:23:26.703 INFO:journalctl@ceph.mon.a.smithi079.stdout: filter_policy: bloomfilter 2024-09-21T12:23:26.703 INFO:journalctl@ceph.mon.a.smithi079.stdout: whole_key_filtering: 1 2024-09-21T12:23:26.704 INFO:journalctl@ceph.mon.a.smithi079.stdout: verify_compression: 0 2024-09-21T12:23:26.704 INFO:journalctl@ceph.mon.a.smithi079.stdout: read_amp_bytes_per_bit: 0 2024-09-21T12:23:26.704 INFO:journalctl@ceph.mon.a.smithi079.stdout: format_version: 5 2024-09-21T12:23:26.704 INFO:journalctl@ceph.mon.a.smithi079.stdout: enable_index_compression: 1 2024-09-21T12:23:26.704 INFO:journalctl@ceph.mon.a.smithi079.stdout: block_align: 0 2024-09-21T12:23:26.704 INFO:journalctl@ceph.mon.a.smithi079.stdout: max_auto_readahead_size: 262144 2024-09-21T12:23:26.704 INFO:journalctl@ceph.mon.a.smithi079.stdout: prepopulate_block_cache: 0 2024-09-21T12:23:26.704 INFO:journalctl@ceph.mon.a.smithi079.stdout: initial_auto_readahead_size: 8192 2024-09-21T12:23:26.704 INFO:journalctl@ceph.mon.a.smithi079.stdout: num_file_reads_for_auto_readahead: 2 2024-09-21T12:23:26.704 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.write_buffer_size: 33554432 2024-09-21T12:23:26.704 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.max_write_buffer_number: 2 2024-09-21T12:23:26.704 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.compression: NoCompression 2024-09-21T12:23:26.704 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.bottommost_compression: Disabled 2024-09-21T12:23:26.704 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.prefix_extractor: nullptr 2024-09-21T12:23:26.704 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-21T12:23:26.705 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.num_levels: 7 2024-09-21T12:23:26.705 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-21T12:23:26.705 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-21T12:23:26.705 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-21T12:23:26.705 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-21T12:23:26.705 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-21T12:23:26.705 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-21T12:23:26.705 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-21T12:23:26.705 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-21T12:23:26.705 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-21T12:23:26.705 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-21T12:23:26.705 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-09-21T12:23:26.705 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-09-21T12:23:26.705 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-21T12:23:26.705 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.compression_opts.level: 32767 2024-09-21T12:23:26.706 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.compression_opts.strategy: 0 2024-09-21T12:23:26.706 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-21T12:23:26.706 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-21T12:23:26.706 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-09-21T12:23:26.706 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-21T12:23:26.706 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.compression_opts.enabled: false 2024-09-21T12:23:26.706 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-09-21T12:23:26.706 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-21T12:23:26.706 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-21T12:23:26.706 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-21T12:23:26.706 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.target_file_size_base: 67108864 2024-09-21T12:23:26.706 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-21T12:23:26.706 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-21T12:23:26.706 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-21T12:23:26.707 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-21T12:23:26.707 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-21T12:23:26.707 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-21T12:23:26.707 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-21T12:23:26.707 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-21T12:23:26.707 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-21T12:23:26.707 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-21T12:23:26.707 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-21T12:23:26.707 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-21T12:23:26.707 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-21T12:23:26.707 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-09-21T12:23:26.707 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.arena_block_size: 1048576 2024-09-21T12:23:26.707 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-21T12:23:26.707 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-21T12:23:26.707 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.disable_auto_compactions: 0 2024-09-21T12:23:26.708 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-21T12:23:26.708 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-21T12:23:26.708 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-21T12:23:26.708 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-21T12:23:26.708 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-21T12:23:26.708 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-21T12:23:26.708 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-21T12:23:26.708 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-21T12:23:26.708 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-21T12:23:26.708 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-21T12:23:26.708 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-21T12:23:26.708 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.inplace_update_support: 0 2024-09-21T12:23:26.708 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-21T12:23:26.708 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-21T12:23:26.708 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-21T12:23:26.709 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-21T12:23:26.709 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.bloom_locality: 0 2024-09-21T12:23:26.709 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.max_successive_merges: 0 2024-09-21T12:23:26.709 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-21T12:23:26.709 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.paranoid_file_checks: 0 2024-09-21T12:23:26.709 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.force_consistency_checks: 1 2024-09-21T12:23:26.709 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.report_bg_io_stats: 0 2024-09-21T12:23:26.709 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.ttl: 2592000 2024-09-21T12:23:26.709 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-21T12:23:26.709 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-09-21T12:23:26.709 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-09-21T12:23:26.709 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.enable_blob_files: false 2024-09-21T12:23:26.709 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.min_blob_size: 0 2024-09-21T12:23:26.709 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.blob_file_size: 268435456 2024-09-21T12:23:26.709 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-21T12:23:26.710 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-21T12:23:26.710 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-21T12:23:26.710 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-09-21T12:23:26.710 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-09-21T12:23:26.710 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.blob_file_starting_level: 0 2024-09-21T12:23:26.710 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-09-21T12:23:26.710 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-09-21T12:23:26.710 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-09-21T12:23:26.710 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 656adc8b-175f-4d81-a889-163512a0e0f1 2024-09-21T12:23:26.710 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726921406679984, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-09-21T12:23:26.710 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-09-21T12:23:26.710 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726921406680806, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 80289, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 252, "table_properties": {"data_size": 78450, "index_size": 223, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 581, "raw_key_size": 10027, "raw_average_key_size": 47, "raw_value_size": 72761, "raw_average_value_size": 344, "num_data_blocks": 10, "num_entries": 211, "num_filter_entries": 211, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1726921406, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "656adc8b-175f-4d81-a889-163512a0e0f1", "db_session_id": "TSDRK3JJIYZ5IQIC7ZHC", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-09-21T12:23:26.710 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726921406680900, "job": 1, "event": "recovery_finished"} 2024-09-21T12:23:26.711 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-09-21T12:23:26.711 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-21T12:23:26.711 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x557c92690e00 2024-09-21T12:23:26.711 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: DB pointer 0x557c92784000 2024-09-21T12:23:26.711 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-09-21T12:23:26.711 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-09-21T12:23:26.711 INFO:journalctl@ceph.mon.a.smithi079.stdout: ** DB Stats ** 2024-09-21T12:23:26.711 INFO:journalctl@ceph.mon.a.smithi079.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-21T12:23:26.711 INFO:journalctl@ceph.mon.a.smithi079.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-21T12:23:26.711 INFO:journalctl@ceph.mon.a.smithi079.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-21T12:23:26.711 INFO:journalctl@ceph.mon.a.smithi079.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-21T12:23:26.711 INFO:journalctl@ceph.mon.a.smithi079.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-21T12:23:26.711 INFO:journalctl@ceph.mon.a.smithi079.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-21T12:23:26.711 INFO:journalctl@ceph.mon.a.smithi079.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-21T12:23:26.711 INFO:journalctl@ceph.mon.a.smithi079.stdout: 2024-09-21T12:23:26.712 INFO:journalctl@ceph.mon.a.smithi079.stdout: ** Compaction Stats [default] ** 2024-09-21T12:23:26.712 INFO:journalctl@ceph.mon.a.smithi079.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-21T12:23:26.712 INFO:journalctl@ceph.mon.a.smithi079.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-09-21T12:23:26.712 INFO:journalctl@ceph.mon.a.smithi079.stdout: L0 2/0 80.27 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 144.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-21T12:23:26.712 INFO:journalctl@ceph.mon.a.smithi079.stdout: Sum 2/0 80.27 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 144.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-21T12:23:26.712 INFO:journalctl@ceph.mon.a.smithi079.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 144.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-21T12:23:26.712 INFO:journalctl@ceph.mon.a.smithi079.stdout: 2024-09-21T12:23:26.712 INFO:journalctl@ceph.mon.a.smithi079.stdout: ** Compaction Stats [default] ** 2024-09-21T12:23:26.712 INFO:journalctl@ceph.mon.a.smithi079.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-21T12:23:26.712 INFO:journalctl@ceph.mon.a.smithi079.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-21T12:23:26.712 INFO:journalctl@ceph.mon.a.smithi079.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 144.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-21T12:23:26.712 INFO:journalctl@ceph.mon.a.smithi079.stdout: 2024-09-21T12:23:26.712 INFO:journalctl@ceph.mon.a.smithi079.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-09-21T12:23:26.712 INFO:journalctl@ceph.mon.a.smithi079.stdout: 2024-09-21T12:23:26.712 INFO:journalctl@ceph.mon.a.smithi079.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-21T12:23:26.713 INFO:journalctl@ceph.mon.a.smithi079.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-21T12:23:26.713 INFO:journalctl@ceph.mon.a.smithi079.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-21T12:23:26.713 INFO:journalctl@ceph.mon.a.smithi079.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-21T12:23:26.713 INFO:journalctl@ceph.mon.a.smithi079.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-21T12:23:26.713 INFO:journalctl@ceph.mon.a.smithi079.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-21T12:23:26.713 INFO:journalctl@ceph.mon.a.smithi079.stdout: Cumulative compaction: 0.00 GB write, 13.41 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-21T12:23:26.713 INFO:journalctl@ceph.mon.a.smithi079.stdout: Interval compaction: 0.00 GB write, 13.41 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-21T12:23:26.713 INFO:journalctl@ceph.mon.a.smithi079.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-21T12:23:26.713 INFO:journalctl@ceph.mon.a.smithi079.stdout: Block cache BinnedLRUCache@0x557c92665350#2 capacity: 512.00 MB usage: 1.17 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-09-21T12:23:26.713 INFO:journalctl@ceph.mon.a.smithi079.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.77 KB,0.000146031%) IndexBlock(2,0.41 KB,7.7486e-05%) Misc(1,0.00 KB,0%) 2024-09-21T12:23:26.713 INFO:journalctl@ceph.mon.a.smithi079.stdout: 2024-09-21T12:23:26.713 INFO:journalctl@ceph.mon.a.smithi079.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-21T12:23:26.713 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: starting mon.a rank 0 at public addrs [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] at bind addrs [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 33168022-7814-11ef-baf6-efdc52797490 2024-09-21T12:23:26.713 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: mon.a@-1(???) e1 preinit fsid 33168022-7814-11ef-baf6-efdc52797490 2024-09-21T12:23:26.714 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: mon.a@-1(???).mds e1 new map 2024-09-21T12:23:26.714 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: mon.a@-1(???).mds e1 print_map 2024-09-21T12:23:26.714 INFO:journalctl@ceph.mon.a.smithi079.stdout: e1 2024-09-21T12:23:26.714 INFO:journalctl@ceph.mon.a.smithi079.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-21T12:23:26.714 INFO:journalctl@ceph.mon.a.smithi079.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-09-21T12:23:26.714 INFO:journalctl@ceph.mon.a.smithi079.stdout: legacy client fscid: -1 2024-09-21T12:23:26.714 INFO:journalctl@ceph.mon.a.smithi079.stdout: 2024-09-21T12:23:26.714 INFO:journalctl@ceph.mon.a.smithi079.stdout: No filesystems configured 2024-09-21T12:23:26.714 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-09-21T12:23:26.714 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-21T12:23:26.714 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-21T12:23:26.714 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-21T12:23:26.714 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-09-21T12:23:26.726 INFO:teuthology.orchestra.run.smithi079.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-09-21T12:23:27.055 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 bash[23680]: 794d66ac29c5f080f3d25c707ee3cb754cd838c130700defb586dd0788bdd96a 2024-09-21T12:23:27.055 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 systemd[1]: Started Ceph mon.a for 33168022-7814-11ef-baf6-efdc52797490. 2024-09-21T12:23:27.055 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: mon.a is new leader, mons a in quorum (ranks 0) 2024-09-21T12:23:27.055 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: monmap e1: 1 mons at {a=[v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-21T12:23:27.055 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: fsmap 2024-09-21T12:23:27.055 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: osdmap e1: 0 total, 0 up, 0 in 2024-09-21T12:23:27.055 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:26 smithi079 ceph-mon[23699]: mgrmap e1: no daemons active 2024-09-21T12:23:27.836 INFO:teuthology.orchestra.run.smithi079.stdout:Wrote config to /etc/ceph/ceph.conf 2024-09-21T12:23:27.858 INFO:teuthology.orchestra.run.smithi079.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-09-21T12:23:27.858 INFO:teuthology.orchestra.run.smithi079.stdout:Creating mgr... 2024-09-21T12:23:27.858 INFO:teuthology.orchestra.run.smithi079.stdout:Verifying port 0.0.0.0:9283 ... 2024-09-21T12:23:27.859 INFO:teuthology.orchestra.run.smithi079.stdout:Verifying port 0.0.0.0:8765 ... 2024-09-21T12:23:28.122 INFO:teuthology.orchestra.run.smithi079.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-33168022-7814-11ef-baf6-efdc52797490@mgr.y 2024-09-21T12:23:28.123 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stderr Failed to reset failed state of unit ceph-33168022-7814-11ef-baf6-efdc52797490@mgr.y.service: Unit ceph-33168022-7814-11ef-baf6-efdc52797490@mgr.y.service not loaded. 2024-09-21T12:23:28.262 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-33168022-7814-11ef-baf6-efdc52797490.target.wants/ceph-33168022-7814-11ef-baf6-efdc52797490@mgr.y.service → /etc/systemd/system/ceph-33168022-7814-11ef-baf6-efdc52797490@.service. 2024-09-21T12:23:28.281 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:23:28 smithi079 systemd[1]: Starting Ceph mgr.y for 33168022-7814-11ef-baf6-efdc52797490... 2024-09-21T12:23:28.589 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:28 smithi079 ceph-mon[23699]: from='client.? 172.21.15.79:0/538684701' entity='client.admin' 2024-09-21T12:23:28.589 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:23:28 smithi079 podman[23909]: 2024-09-21 12:23:28.390690842 +0000 UTC m=+0.015617248 image pull 0adab276c97e6f3143ffbca5eeff37fcd7dc3b5c0f339a9f747381d9993602de quay.ceph.io/ceph-ci/ceph:reef 2024-09-21T12:23:28.589 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:23:28 smithi079 podman[23909]: 2024-09-21 12:23:28.493150754 +0000 UTC m=+0.118077146 container create e34c6cf8a73d28c0780d22cef3c05911324af321edfdd4de57c877e59e1e16ad (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y, org.label-schema.build-date=20240731, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, RELEASE=reef-6a1bdf8, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True) 2024-09-21T12:23:28.802 INFO:teuthology.orchestra.run.smithi079.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-21T12:23:28.802 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stdout disabled 2024-09-21T12:23:28.809 INFO:teuthology.orchestra.run.smithi079.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-21T12:23:28.809 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stdout inactive 2024-09-21T12:23:28.809 INFO:teuthology.orchestra.run.smithi079.stdout:firewalld.service is not enabled 2024-09-21T12:23:28.809 INFO:teuthology.orchestra.run.smithi079.stdout:Not possible to enable service . firewalld.service is not available 2024-09-21T12:23:28.817 INFO:teuthology.orchestra.run.smithi079.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-21T12:23:28.817 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stdout disabled 2024-09-21T12:23:28.825 INFO:teuthology.orchestra.run.smithi079.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-21T12:23:28.825 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stdout inactive 2024-09-21T12:23:28.825 INFO:teuthology.orchestra.run.smithi079.stdout:firewalld.service is not enabled 2024-09-21T12:23:28.825 INFO:teuthology.orchestra.run.smithi079.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-09-21T12:23:28.825 INFO:teuthology.orchestra.run.smithi079.stdout:Waiting for mgr to start... 2024-09-21T12:23:28.825 INFO:teuthology.orchestra.run.smithi079.stdout:Waiting for mgr... 2024-09-21T12:23:28.870 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:23:28 smithi079 podman[23909]: 2024-09-21 12:23:28.687706878 +0000 UTC m=+0.312633269 container init e34c6cf8a73d28c0780d22cef3c05911324af321edfdd4de57c877e59e1e16ad (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y, RELEASE=reef-6a1bdf8, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T12:23:28.870 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:23:28 smithi079 podman[23909]: 2024-09-21 12:23:28.690565681 +0000 UTC m=+0.315492075 container start e34c6cf8a73d28c0780d22cef3c05911324af321edfdd4de57c877e59e1e16ad (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, RELEASE=reef-6a1bdf8, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-09-21T12:23:28.871 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:23:28 smithi079 bash[23909]: e34c6cf8a73d28c0780d22cef3c05911324af321edfdd4de57c877e59e1e16ad 2024-09-21T12:23:28.871 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:23:28 smithi079 systemd[1]: Started Ceph mgr.y for 33168022-7814-11ef-baf6-efdc52797490. 2024-09-21T12:23:28.871 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:23:28 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: 2024-09-21T12:23:28.808+0000 7f0f95bf8040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-21T12:23:28.871 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:23:28 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: 2024-09-21T12:23:28.868+0000 7f0f95bf8040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-21T12:23:29.167 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:23:29 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: 2024-09-21T12:23:29.017+0000 7f0f95bf8040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-21T12:23:29.167 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:23:29 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: 2024-09-21T12:23:29.086+0000 7f0f95bf8040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-21T12:23:29.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:29 smithi079 ceph-mon[23699]: from='client.? 172.21.15.79:0/1817003241' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-09-21T12:23:29.442 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:23:29 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: 2024-09-21T12:23:29.235+0000 7f0f95bf8040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-21T12:23:29.442 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:23:29 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: 2024-09-21T12:23:29.391+0000 7f0f95bf8040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-21T12:23:29.821 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:23:29 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: 2024-09-21T12:23:29.704+0000 7f0f95bf8040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-21T12:23:29.821 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:23:29 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: 2024-09-21T12:23:29.766+0000 7f0f95bf8040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-21T12:23:29.825 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout 2024-09-21T12:23:29.825 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout { 2024-09-21T12:23:29.825 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "fsid": "33168022-7814-11ef-baf6-efdc52797490", 2024-09-21T12:23:29.825 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "health": { 2024-09-21T12:23:29.826 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-21T12:23:29.826 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-21T12:23:29.826 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-21T12:23:29.826 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-09-21T12:23:29.826 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-21T12:23:29.826 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-21T12:23:29.826 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout 0 2024-09-21T12:23:29.826 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout ], 2024-09-21T12:23:29.826 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-21T12:23:29.826 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "a" 2024-09-21T12:23:29.826 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout ], 2024-09-21T12:23:29.826 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-09-21T12:23:29.826 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-21T12:23:29.826 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T12:23:29.826 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2024-09-21T12:23:29.827 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-21T12:23:29.827 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-09-21T12:23:29.827 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-21T12:23:29.827 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T12:23:29.827 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-21T12:23:29.827 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-21T12:23:29.827 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-21T12:23:29.827 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-21T12:23:29.830 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-21T12:23:29.830 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-21T12:23:29.830 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-09-21T12:23:29.830 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-21T12:23:29.830 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-21T12:23:29.830 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-21T12:23:29.830 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-21T12:23:29.831 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-21T12:23:29.831 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-21T12:23:29.831 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-21T12:23:29.831 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-21T12:23:29.831 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-21T12:23:29.831 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-09-21T12:23:29.831 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-21T12:23:29.831 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T12:23:29.831 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-21T12:23:29.831 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-21T12:23:29.831 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-09-21T12:23:29.831 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-21T12:23:29.831 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "available": false, 2024-09-21T12:23:29.831 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-21T12:23:29.832 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-21T12:23:29.832 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-21T12:23:29.832 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-21T12:23:29.832 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "restful" 2024-09-21T12:23:29.832 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout ], 2024-09-21T12:23:29.832 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-21T12:23:29.832 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-09-21T12:23:29.832 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-21T12:23:29.832 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T12:23:29.832 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "modified": "2024-09-21T12:23:21.897000+0000", 2024-09-21T12:23:29.832 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-21T12:23:29.832 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-09-21T12:23:29.832 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-21T12:23:29.832 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout } 2024-09-21T12:23:29.832 INFO:teuthology.orchestra.run.smithi079.stdout:mgr not available, waiting (1/15)... 2024-09-21T12:23:30.192 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:23:29 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: 2024-09-21T12:23:29.948+0000 7f0f95bf8040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-21T12:23:31.192 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:23:30 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: 2024-09-21T12:23:30.800+0000 7f0f95bf8040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-21T12:23:31.192 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:23:30 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: 2024-09-21T12:23:30.865+0000 7f0f95bf8040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-21T12:23:31.884 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:23:31 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: 2024-09-21T12:23:31.569+0000 7f0f95bf8040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-21T12:23:31.884 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:23:31 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: 2024-09-21T12:23:31.680+0000 7f0f95bf8040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-21T12:23:31.884 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:23:31 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: 2024-09-21T12:23:31.818+0000 7f0f95bf8040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-21T12:23:31.884 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:23:31 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: 2024-09-21T12:23:31.882+0000 7f0f95bf8040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-21T12:23:32.160 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:23:32 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: 2024-09-21T12:23:32.029+0000 7f0f95bf8040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-21T12:23:32.160 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:23:32 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: 2024-09-21T12:23:32.098+0000 7f0f95bf8040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-21T12:23:32.160 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:23:32 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: 2024-09-21T12:23:32.158+0000 7f0f95bf8040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-21T12:23:32.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:32 smithi079 ceph-mon[23699]: from='client.? 172.21.15.79:0/3433717107' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-09-21T12:23:32.442 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:23:32 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: 2024-09-21T12:23:32.319+0000 7f0f95bf8040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-21T12:23:32.442 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:23:32 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: 2024-09-21T12:23:32.387+0000 7f0f95bf8040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-21T12:23:32.819 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:23:32 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: 2024-09-21T12:23:32.453+0000 7f0f95bf8040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-21T12:23:32.820 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:23:32 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: 2024-09-21T12:23:32.535+0000 7f0f95bf8040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-21T12:23:32.820 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:23:32 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: 2024-09-21T12:23:32.596+0000 7f0f95bf8040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-21T12:23:32.820 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:23:32 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: 2024-09-21T12:23:32.656+0000 7f0f95bf8040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-21T12:23:32.827 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout 2024-09-21T12:23:32.827 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout { 2024-09-21T12:23:32.827 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "fsid": "33168022-7814-11ef-baf6-efdc52797490", 2024-09-21T12:23:32.827 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "health": { 2024-09-21T12:23:32.827 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-21T12:23:32.827 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-21T12:23:32.827 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-21T12:23:32.827 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-09-21T12:23:32.827 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-21T12:23:32.827 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-21T12:23:32.827 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout 0 2024-09-21T12:23:32.827 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout ], 2024-09-21T12:23:32.828 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-21T12:23:32.828 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "a" 2024-09-21T12:23:32.828 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout ], 2024-09-21T12:23:32.828 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2024-09-21T12:23:32.828 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-21T12:23:32.828 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T12:23:32.828 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2024-09-21T12:23:32.828 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-21T12:23:32.828 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-09-21T12:23:32.828 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-21T12:23:32.828 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T12:23:32.828 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-21T12:23:32.828 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-21T12:23:32.828 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-21T12:23:32.828 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-21T12:23:32.829 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-21T12:23:32.829 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-21T12:23:32.829 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-09-21T12:23:32.829 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-21T12:23:32.829 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-21T12:23:32.829 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-21T12:23:32.829 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-21T12:23:32.829 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-21T12:23:32.829 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-21T12:23:32.829 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-21T12:23:32.829 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-21T12:23:32.829 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-21T12:23:32.830 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-09-21T12:23:32.830 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-21T12:23:32.830 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T12:23:32.830 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-21T12:23:32.830 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-21T12:23:32.830 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-09-21T12:23:32.830 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-21T12:23:32.830 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "available": false, 2024-09-21T12:23:32.830 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-21T12:23:32.830 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-21T12:23:32.830 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-21T12:23:32.830 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-21T12:23:32.830 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "restful" 2024-09-21T12:23:32.830 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout ], 2024-09-21T12:23:32.830 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-21T12:23:32.831 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-09-21T12:23:32.831 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-21T12:23:32.831 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T12:23:32.831 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "modified": "2024-09-21T12:23:21.897000+0000", 2024-09-21T12:23:32.831 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-21T12:23:32.831 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-09-21T12:23:32.831 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-21T12:23:32.831 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout } 2024-09-21T12:23:32.831 INFO:teuthology.orchestra.run.smithi079.stdout:mgr not available, waiting (2/15)... 2024-09-21T12:23:33.942 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:23:33 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: 2024-09-21T12:23:33.441+0000 7f0f95bf8040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-21T12:23:33.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:33 smithi079 ceph-mon[23699]: Activating manager daemon y 2024-09-21T12:23:33.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:33 smithi079 ceph-mon[23699]: mgrmap e2: y(active, starting, since 0.00282384s) 2024-09-21T12:23:33.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:33 smithi079 ceph-mon[23699]: from='mgr.14100 172.21.15.79:0/3607249870' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-21T12:23:33.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:33 smithi079 ceph-mon[23699]: from='mgr.14100 172.21.15.79:0/3607249870' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-21T12:23:33.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:33 smithi079 ceph-mon[23699]: from='mgr.14100 172.21.15.79:0/3607249870' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-21T12:23:33.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:33 smithi079 ceph-mon[23699]: from='mgr.14100 172.21.15.79:0/3607249870' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T12:23:33.943 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:33 smithi079 ceph-mon[23699]: from='mgr.14100 172.21.15.79:0/3607249870' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-21T12:23:33.943 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:33 smithi079 ceph-mon[23699]: Manager daemon y is now available 2024-09-21T12:23:33.943 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:33 smithi079 ceph-mon[23699]: from='mgr.14100 172.21.15.79:0/3607249870' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-21T12:23:33.943 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:33 smithi079 ceph-mon[23699]: from='mgr.14100 172.21.15.79:0/3607249870' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-21T12:23:33.943 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:33 smithi079 ceph-mon[23699]: from='mgr.14100 172.21.15.79:0/3607249870' entity='mgr.y' 2024-09-21T12:23:33.943 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:33 smithi079 ceph-mon[23699]: from='mgr.14100 172.21.15.79:0/3607249870' entity='mgr.y' 2024-09-21T12:23:33.943 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:33 smithi079 ceph-mon[23699]: from='mgr.14100 172.21.15.79:0/3607249870' entity='mgr.y' 2024-09-21T12:23:35.691 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:35 smithi079 ceph-mon[23699]: mgrmap e3: y(active, since 1.0068s) 2024-09-21T12:23:35.964 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout 2024-09-21T12:23:35.964 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout { 2024-09-21T12:23:35.964 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "fsid": "33168022-7814-11ef-baf6-efdc52797490", 2024-09-21T12:23:35.964 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "health": { 2024-09-21T12:23:35.964 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-21T12:23:35.964 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-21T12:23:35.965 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-21T12:23:35.965 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-09-21T12:23:35.965 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-21T12:23:35.965 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-21T12:23:35.965 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout 0 2024-09-21T12:23:35.965 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout ], 2024-09-21T12:23:35.965 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-21T12:23:35.965 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "a" 2024-09-21T12:23:35.965 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout ], 2024-09-21T12:23:35.965 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "quorum_age": 8, 2024-09-21T12:23:35.965 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-21T12:23:35.965 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T12:23:35.965 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2024-09-21T12:23:35.965 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-21T12:23:35.966 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-09-21T12:23:35.966 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-21T12:23:35.966 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T12:23:35.966 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-21T12:23:35.966 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-21T12:23:35.966 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-21T12:23:35.966 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-21T12:23:35.969 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-21T12:23:35.970 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-21T12:23:35.970 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-09-21T12:23:35.970 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-21T12:23:35.970 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-21T12:23:35.970 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-21T12:23:35.970 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-21T12:23:35.970 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-21T12:23:35.970 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-21T12:23:35.970 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-21T12:23:35.970 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-21T12:23:35.970 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-21T12:23:35.970 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-09-21T12:23:35.971 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-21T12:23:35.971 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T12:23:35.971 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-21T12:23:35.971 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-21T12:23:35.971 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-09-21T12:23:35.971 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-21T12:23:35.972 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-21T12:23:35.972 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-21T12:23:35.972 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-21T12:23:35.972 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-21T12:23:35.972 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-21T12:23:35.972 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "restful" 2024-09-21T12:23:35.972 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout ], 2024-09-21T12:23:35.972 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-21T12:23:35.972 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-09-21T12:23:35.972 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-21T12:23:35.972 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T12:23:35.973 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "modified": "2024-09-21T12:23:21.897000+0000", 2024-09-21T12:23:35.973 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-21T12:23:35.973 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout }, 2024-09-21T12:23:35.973 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-21T12:23:35.973 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout } 2024-09-21T12:23:35.973 INFO:teuthology.orchestra.run.smithi079.stdout:mgr is available 2024-09-21T12:23:36.497 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:36 smithi079 ceph-mon[23699]: mgrmap e4: y(active, since 2s) 2024-09-21T12:23:36.497 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:36 smithi079 ceph-mon[23699]: from='client.? 172.21.15.79:0/1442788810' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-09-21T12:23:37.160 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout 2024-09-21T12:23:37.160 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout [global] 2024-09-21T12:23:37.160 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout fsid = 33168022-7814-11ef-baf6-efdc52797490 2024-09-21T12:23:37.160 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-09-21T12:23:37.160 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-09-21T12:23:37.160 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-09-21T12:23:37.160 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-09-21T12:23:37.160 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout 2024-09-21T12:23:37.160 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout [mgr] 2024-09-21T12:23:37.160 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-09-21T12:23:37.160 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout 2024-09-21T12:23:37.160 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout [osd] 2024-09-21T12:23:37.160 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-09-21T12:23:37.161 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-09-21T12:23:37.161 INFO:teuthology.orchestra.run.smithi079.stdout:Enabling cephadm module... 2024-09-21T12:23:37.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:37 smithi079 ceph-mon[23699]: from='client.? 172.21.15.79:0/3112479166' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-09-21T12:23:37.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:37 smithi079 ceph-mon[23699]: from='client.? 172.21.15.79:0/3112479166' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-09-21T12:23:40.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:40 smithi079 ceph-mon[23699]: from='client.? 172.21.15.79:0/3777841911' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-09-21T12:23:40.942 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:23:40 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: ignoring --setuser ceph since I am not root 2024-09-21T12:23:40.942 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:23:40 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: ignoring --setgroup ceph since I am not root 2024-09-21T12:23:40.942 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:23:40 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: 2024-09-21T12:23:40.803+0000 7f2c9e985040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-21T12:23:40.942 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:23:40 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: 2024-09-21T12:23:40.858+0000 7f2c9e985040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-21T12:23:41.355 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:23:41 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: 2024-09-21T12:23:41.005+0000 7f2c9e985040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-21T12:23:41.355 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:23:41 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: 2024-09-21T12:23:41.071+0000 7f2c9e985040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-21T12:23:41.355 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:23:41 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: 2024-09-21T12:23:41.213+0000 7f2c9e985040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-21T12:23:41.355 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:23:41 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: 2024-09-21T12:23:41.353+0000 7f2c9e985040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-21T12:23:41.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:41 smithi079 ceph-mon[23699]: from='client.? 172.21.15.79:0/3777841911' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-09-21T12:23:41.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:41 smithi079 ceph-mon[23699]: mgrmap e5: y(active, since 7s) 2024-09-21T12:23:41.942 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:23:41 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: 2024-09-21T12:23:41.657+0000 7f2c9e985040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-21T12:23:41.942 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:23:41 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: 2024-09-21T12:23:41.718+0000 7f2c9e985040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-21T12:23:41.942 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:23:41 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: 2024-09-21T12:23:41.898+0000 7f2c9e985040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-21T12:23:43.082 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:23:42 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: 2024-09-21T12:23:42.751+0000 7f2c9e985040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-21T12:23:43.082 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:23:42 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: 2024-09-21T12:23:42.817+0000 7f2c9e985040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-21T12:23:43.815 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:23:43 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: 2024-09-21T12:23:43.552+0000 7f2c9e985040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-21T12:23:43.815 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:23:43 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: 2024-09-21T12:23:43.668+0000 7f2c9e985040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-21T12:23:44.093 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:23:43 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: 2024-09-21T12:23:43.812+0000 7f2c9e985040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-21T12:23:44.093 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:23:43 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: 2024-09-21T12:23:43.874+0000 7f2c9e985040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-21T12:23:44.093 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:23:44 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: 2024-09-21T12:23:44.022+0000 7f2c9e985040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-21T12:23:44.093 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:23:44 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: 2024-09-21T12:23:44.091+0000 7f2c9e985040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-21T12:23:44.372 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:23:44 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: 2024-09-21T12:23:44.150+0000 7f2c9e985040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-21T12:23:44.372 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:23:44 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: 2024-09-21T12:23:44.307+0000 7f2c9e985040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-21T12:23:44.372 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:23:44 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: 2024-09-21T12:23:44.370+0000 7f2c9e985040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-21T12:23:44.636 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:23:44 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: 2024-09-21T12:23:44.436+0000 7f2c9e985040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-21T12:23:44.636 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:23:44 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: 2024-09-21T12:23:44.517+0000 7f2c9e985040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-21T12:23:44.636 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:23:44 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: 2024-09-21T12:23:44.575+0000 7f2c9e985040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-21T12:23:44.636 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:23:44 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: 2024-09-21T12:23:44.633+0000 7f2c9e985040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-21T12:23:45.692 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:23:45 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: 2024-09-21T12:23:45.403+0000 7f2c9e985040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-21T12:23:45.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:45 smithi079 ceph-mon[23699]: Active manager daemon y restarted 2024-09-21T12:23:45.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:45 smithi079 ceph-mon[23699]: Activating manager daemon y 2024-09-21T12:23:45.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:45 smithi079 ceph-mon[23699]: osdmap e2: 0 total, 0 up, 0 in 2024-09-21T12:23:45.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:45 smithi079 ceph-mon[23699]: mgrmap e6: y(active, starting, since 0.00317308s) 2024-09-21T12:23:45.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:45 smithi079 ceph-mon[23699]: from='mgr.14118 172.21.15.79:0/613404073' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T12:23:45.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:45 smithi079 ceph-mon[23699]: from='mgr.14118 172.21.15.79:0/613404073' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-21T12:23:45.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:45 smithi079 ceph-mon[23699]: from='mgr.14118 172.21.15.79:0/613404073' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-21T12:23:45.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:45 smithi079 ceph-mon[23699]: from='mgr.14118 172.21.15.79:0/613404073' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-21T12:23:45.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:45 smithi079 ceph-mon[23699]: from='mgr.14118 172.21.15.79:0/613404073' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-21T12:23:45.693 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:45 smithi079 ceph-mon[23699]: Manager daemon y is now available 2024-09-21T12:23:45.693 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:45 smithi079 ceph-mon[23699]: from='mgr.14118 172.21.15.79:0/613404073' entity='mgr.y' 2024-09-21T12:23:45.693 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:45 smithi079 ceph-mon[23699]: from='mgr.14118 172.21.15.79:0/613404073' entity='mgr.y' 2024-09-21T12:23:45.693 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:45 smithi079 ceph-mon[23699]: from='mgr.14118 172.21.15.79:0/613404073' entity='mgr.y' 2024-09-21T12:23:45.693 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:45 smithi079 ceph-mon[23699]: from='mgr.14118 172.21.15.79:0/613404073' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:23:45.693 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:45 smithi079 ceph-mon[23699]: from='mgr.14118 172.21.15.79:0/613404073' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:23:45.693 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:45 smithi079 ceph-mon[23699]: from='mgr.14118 172.21.15.79:0/613404073' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-21T12:23:45.693 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:45 smithi079 ceph-mon[23699]: from='mgr.14118 172.21.15.79:0/613404073' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-21T12:23:46.838 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:46 smithi079 ceph-mon[23699]: Found migration_current of "None". Setting to last migration. 2024-09-21T12:23:46.838 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:46 smithi079 ceph-mon[23699]: from='mgr.14118 172.21.15.79:0/613404073' entity='mgr.y' 2024-09-21T12:23:46.838 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:46 smithi079 ceph-mon[23699]: from='mgr.14118 172.21.15.79:0/613404073' entity='mgr.y' 2024-09-21T12:23:46.838 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:46 smithi079 ceph-mon[23699]: mgrmap e7: y(active, since 1.00565s) 2024-09-21T12:23:47.140 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:23:46 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: 2024-09-21T12:23:46.836+0000 7f2c06665640 -1 log_channel(cephadm) log [ERR] : [21/Sep/2024:12:23:46] ENGINE Error in HTTPServer.serve 2024-09-21T12:23:47.140 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:23:46 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: Traceback (most recent call last): 2024-09-21T12:23:47.140 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:23:46 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T12:23:47.141 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:23:46 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: self._connections.run(self.expiration_interval) 2024-09-21T12:23:47.141 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:23:46 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T12:23:47.141 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:23:46 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: self._run(expiration_interval) 2024-09-21T12:23:47.141 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:23:46 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T12:23:47.141 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:23:46 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T12:23:47.141 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:23:46 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T12:23:47.141 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:23:46 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T12:23:47.141 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:23:46 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T12:23:47.141 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:23:46 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: s = self.context.wrap_socket( 2024-09-21T12:23:47.141 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:23:46 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T12:23:47.142 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:23:46 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: return self.sslsocket_class._create( 2024-09-21T12:23:47.142 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:23:46 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T12:23:47.142 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:23:46 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: self.do_handshake() 2024-09-21T12:23:47.142 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:23:46 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T12:23:47.142 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:23:46 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: self._sslobj.do_handshake() 2024-09-21T12:23:47.142 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:23:46 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T12:23:47.142 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:23:46 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: 2024-09-21T12:23:47.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:47 smithi079 ceph-mon[23699]: [21/Sep/2024:12:23:46] ENGINE Bus STARTING 2024-09-21T12:23:47.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:47 smithi079 ceph-mon[23699]: from='mgr.14118 172.21.15.79:0/613404073' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:23:48.566 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:48 smithi079 ceph-mon[23699]: [21/Sep/2024:12:23:46] ENGINE Serving on http://172.21.15.79:8765 2024-09-21T12:23:48.566 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:48 smithi079 ceph-mon[23699]: [21/Sep/2024:12:23:46] ENGINE Serving on https://172.21.15.79:7150 2024-09-21T12:23:48.566 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:48 smithi079 ceph-mon[23699]: [21/Sep/2024:12:23:46] ENGINE Bus STARTED 2024-09-21T12:23:48.566 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:48 smithi079 ceph-mon[23699]: [21/Sep/2024:12:23:46] ENGINE Error in HTTPServer.serve 2024-09-21T12:23:48.566 INFO:journalctl@ceph.mon.a.smithi079.stdout: Traceback (most recent call last): 2024-09-21T12:23:48.566 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T12:23:48.566 INFO:journalctl@ceph.mon.a.smithi079.stdout: self._connections.run(self.expiration_interval) 2024-09-21T12:23:48.566 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T12:23:48.567 INFO:journalctl@ceph.mon.a.smithi079.stdout: self._run(expiration_interval) 2024-09-21T12:23:48.567 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T12:23:48.567 INFO:journalctl@ceph.mon.a.smithi079.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T12:23:48.567 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T12:23:48.567 INFO:journalctl@ceph.mon.a.smithi079.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T12:23:48.567 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T12:23:48.567 INFO:journalctl@ceph.mon.a.smithi079.stdout: s = self.context.wrap_socket( 2024-09-21T12:23:48.567 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T12:23:48.567 INFO:journalctl@ceph.mon.a.smithi079.stdout: return self.sslsocket_class._create( 2024-09-21T12:23:48.567 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T12:23:48.567 INFO:journalctl@ceph.mon.a.smithi079.stdout: self.do_handshake() 2024-09-21T12:23:48.567 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T12:23:48.567 INFO:journalctl@ceph.mon.a.smithi079.stdout: self._sslobj.do_handshake() 2024-09-21T12:23:48.567 INFO:journalctl@ceph.mon.a.smithi079.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T12:23:48.567 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:48 smithi079 ceph-mon[23699]: mgrmap e8: y(active, since 2s) 2024-09-21T12:23:49.545 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:49 smithi079 ceph-mon[23699]: from='client.? 172.21.15.79:0/1612356785' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-09-21T12:23:49.548 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout { 2024-09-21T12:23:49.548 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "epoch": 8, 2024-09-21T12:23:49.548 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-21T12:23:49.548 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-09-21T12:23:49.548 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-09-21T12:23:49.548 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout } 2024-09-21T12:23:49.548 INFO:teuthology.orchestra.run.smithi079.stdout:Waiting for the mgr to restart... 2024-09-21T12:23:49.548 INFO:teuthology.orchestra.run.smithi079.stdout:Waiting for mgr epoch 8... 2024-09-21T12:23:50.537 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout { 2024-09-21T12:23:50.537 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 8, 2024-09-21T12:23:50.538 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "initialized": true 2024-09-21T12:23:50.538 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout } 2024-09-21T12:23:50.538 INFO:teuthology.orchestra.run.smithi079.stdout:mgr epoch 8 is available 2024-09-21T12:23:50.538 INFO:teuthology.orchestra.run.smithi079.stdout:Setting orchestrator backend to cephadm... 2024-09-21T12:23:50.874 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:50 smithi079 ceph-mon[23699]: from='client.14128 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-09-21T12:23:50.874 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:50 smithi079 ceph-mon[23699]: from='client.14128 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-09-21T12:23:52.237 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:52 smithi079 ceph-mon[23699]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:23:52.237 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:52 smithi079 ceph-mon[23699]: from='mgr.14118 172.21.15.79:0/613404073' entity='mgr.y' 2024-09-21T12:23:52.237 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:52 smithi079 ceph-mon[23699]: from='mgr.14118 172.21.15.79:0/613404073' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:23:52.659 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout value unchanged 2024-09-21T12:23:52.660 INFO:teuthology.orchestra.run.smithi079.stdout:Generating ssh key... 2024-09-21T12:23:53.242 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:53 smithi079 ceph-mon[23699]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:23:53.527 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:23:53 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: Generating public/private rsa key pair. 2024-09-21T12:23:53.527 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:23:53 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: Your identification has been saved in /tmp/tmpxtehghsu/key 2024-09-21T12:23:53.527 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:23:53 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: Your public key has been saved in /tmp/tmpxtehghsu/key.pub 2024-09-21T12:23:53.528 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:23:53 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: The key fingerprint is: 2024-09-21T12:23:53.528 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:23:53 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: SHA256:hmwFsShtTLNk6nhqHIcXjcFMCTvWiPCEWzIsU4nnzhw ceph-33168022-7814-11ef-baf6-efdc52797490 2024-09-21T12:23:53.528 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:23:53 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: The key's randomart image is: 2024-09-21T12:23:53.528 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:23:53 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: +---[RSA 3072]----+ 2024-09-21T12:23:53.528 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:23:53 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: |o=B+* o. | 2024-09-21T12:23:53.528 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:23:53 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: |X+B@++ o | 2024-09-21T12:23:53.528 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:23:53 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: |o&++*.. . | 2024-09-21T12:23:53.528 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:23:53 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: |ooEo.. o | 2024-09-21T12:23:53.528 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:23:53 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: |.*o+ + S | 2024-09-21T12:23:53.528 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:23:53 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: |.oB . . | 2024-09-21T12:23:53.528 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:23:53 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: |.o | 2024-09-21T12:23:53.528 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:23:53 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: |. | 2024-09-21T12:23:53.528 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:23:53 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: | | 2024-09-21T12:23:53.528 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:23:53 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: +----[SHA256]-----+ 2024-09-21T12:23:54.428 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:54 smithi079 ceph-mon[23699]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:23:54.428 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:54 smithi079 ceph-mon[23699]: Generating ssh key... 2024-09-21T12:23:54.428 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:54 smithi079 ceph-mon[23699]: from='mgr.14118 172.21.15.79:0/613404073' entity='mgr.y' 2024-09-21T12:23:54.428 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:54 smithi079 ceph-mon[23699]: from='mgr.14118 172.21.15.79:0/613404073' entity='mgr.y' 2024-09-21T12:23:54.940 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQChPe7ps6ciUCQf0G1KohygC2ZQg803O4LrtJD51v3f8R62etuXbYv7dh5O5jSYrbex4BkCxItIPTukYylXqGgLjHrhPtCIDDmLQipr929jXTMSyIpYxuzeDZuCzlyuYMRuwzV3OX48O5GWrodAQGKs/agdKncpD+fxkqTGNzy90wK1LcKkzowZHdQIhsi7oxDQQ0M+ffOQH3glGovHV118YieCBWE9vO8F3W6QSIuOr62fv3IPeXc4aEtlHtpEEN3LOJei5QuV+MoRBjJgrV/NC8NHe5YkCO8Mk5x3dFGZiwOGM/qoHAlrJYpLdDpzCpijSHtsfMPpKatWeX1oLhhgdDeSwQsdY6KRCd+bIrHZ38bcJviuIh3k6i49OEsQ9csmXWIiZQDVpEGXBSzUPIUIcQIKf3bede2yme1DN1n1r9bVqtoY3Yf4W1cdwE7kIzsJjhJFO7hSDNK2Tqx5+fgXnPAiNA+CI/ZU27jx2k0tMoSxl0uB+UmOv5/r/463I+0= ceph-33168022-7814-11ef-baf6-efdc52797490 2024-09-21T12:23:54.941 INFO:teuthology.orchestra.run.smithi079.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-09-21T12:23:54.941 INFO:teuthology.orchestra.run.smithi079.stdout:Adding key to root@localhost authorized_keys... 2024-09-21T12:23:54.946 INFO:teuthology.orchestra.run.smithi079.stdout:Adding host smithi079... 2024-09-21T12:23:55.291 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:55 smithi079 ceph-mon[23699]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:23:56.647 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:56 smithi079 ceph-mon[23699]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi079", "addr": "172.21.15.79", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:23:57.529 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:57 smithi079 ceph-mon[23699]: Deploying cephadm binary to smithi079 2024-09-21T12:23:59.256 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout Added host 'smithi079' with addr '172.21.15.79' 2024-09-21T12:23:59.256 INFO:teuthology.orchestra.run.smithi079.stdout:Deploying unmanaged mon service... 2024-09-21T12:23:59.362 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:59 smithi079 ceph-mon[23699]: from='mgr.14118 172.21.15.79:0/613404073' entity='mgr.y' 2024-09-21T12:23:59.362 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:59 smithi079 ceph-mon[23699]: Added host smithi079 2024-09-21T12:23:59.362 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:23:59 smithi079 ceph-mon[23699]: from='mgr.14118 172.21.15.79:0/613404073' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:24:00.797 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-09-21T12:24:00.797 INFO:teuthology.orchestra.run.smithi079.stdout:Deploying unmanaged mgr service... 2024-09-21T12:24:01.089 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:00 smithi079 ceph-mon[23699]: from='mgr.14118 172.21.15.79:0/613404073' entity='mgr.y' 2024-09-21T12:24:01.089 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:00 smithi079 ceph-mon[23699]: from='mgr.14118 172.21.15.79:0/613404073' entity='mgr.y' 2024-09-21T12:24:01.089 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:00 smithi079 ceph-mon[23699]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:24:01.089 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:00 smithi079 ceph-mon[23699]: Saving service mon spec with placement count:5 2024-09-21T12:24:01.089 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:00 smithi079 ceph-mon[23699]: from='mgr.14118 172.21.15.79:0/613404073' entity='mgr.y' 2024-09-21T12:24:02.163 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-09-21T12:24:02.615 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:02 smithi079 ceph-mon[23699]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:24:02.616 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:02 smithi079 ceph-mon[23699]: Saving service mgr spec with placement count:2 2024-09-21T12:24:02.616 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:02 smithi079 ceph-mon[23699]: from='mgr.14118 172.21.15.79:0/613404073' entity='mgr.y' 2024-09-21T12:24:03.771 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:03 smithi079 ceph-mon[23699]: from='client.? 172.21.15.79:0/1095466734' entity='client.admin' 2024-09-21T12:24:03.772 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:03 smithi079 ceph-mon[23699]: from='mgr.14118 172.21.15.79:0/613404073' entity='mgr.y' 2024-09-21T12:24:03.772 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:03 smithi079 ceph-mon[23699]: from='mgr.14118 172.21.15.79:0/613404073' entity='mgr.y' 2024-09-21T12:24:03.772 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:03 smithi079 ceph-mon[23699]: from='mgr.14118 172.21.15.79:0/613404073' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi079", "name": "osd_memory_target"}]: dispatch 2024-09-21T12:24:03.772 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:03 smithi079 ceph-mon[23699]: from='mgr.14118 172.21.15.79:0/613404073' entity='mgr.y' 2024-09-21T12:24:03.772 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:03 smithi079 ceph-mon[23699]: from='mgr.14118 172.21.15.79:0/613404073' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi079", "caps": []}]: dispatch 2024-09-21T12:24:03.772 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:03 smithi079 ceph-mon[23699]: from='mgr.14118 172.21.15.79:0/613404073' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.agent.smithi079", "caps": []}]': finished 2024-09-21T12:24:04.561 INFO:teuthology.orchestra.run.smithi079.stdout:Enabling the dashboard module... 2024-09-21T12:24:05.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:04 smithi079 ceph-mon[23699]: from='client.? 172.21.15.79:0/3119714115' entity='client.admin' 2024-09-21T12:24:05.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:04 smithi079 ceph-mon[23699]: Deploying daemon agent.smithi079 on smithi079 2024-09-21T12:24:06.264 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:05 smithi079 ceph-mon[23699]: from='client.? 172.21.15.79:0/942084649' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-09-21T12:24:06.264 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:05 smithi079 ceph-mon[23699]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:24:06.265 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:24:06 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: ignoring --setuser ceph since I am not root 2024-09-21T12:24:06.265 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:24:06 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: ignoring --setgroup ceph since I am not root 2024-09-21T12:24:06.265 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:24:06 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: 2024-09-21T12:24:06.118+0000 7f0e719e5040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-21T12:24:06.265 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:24:06 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: 2024-09-21T12:24:06.174+0000 7f0e719e5040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-21T12:24:06.515 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:24:06 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: 2024-09-21T12:24:06.323+0000 7f0e719e5040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-21T12:24:06.515 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:24:06 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: 2024-09-21T12:24:06.388+0000 7f0e719e5040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-21T12:24:06.767 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:24:06 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: 2024-09-21T12:24:06.537+0000 7f0e719e5040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-21T12:24:06.767 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:24:06 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: 2024-09-21T12:24:06.679+0000 7f0e719e5040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-21T12:24:07.064 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:24:07 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: 2024-09-21T12:24:07.000+0000 7f0e719e5040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-21T12:24:07.064 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:06 smithi079 ceph-mon[23699]: from='client.? 172.21.15.79:0/942084649' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-09-21T12:24:07.064 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:06 smithi079 ceph-mon[23699]: mgrmap e9: y(active, since 20s) 2024-09-21T12:24:07.441 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:24:07 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: 2024-09-21T12:24:07.062+0000 7f0e719e5040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-21T12:24:07.442 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:24:07 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: 2024-09-21T12:24:07.251+0000 7f0e719e5040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-21T12:24:07.750 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout { 2024-09-21T12:24:07.750 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-09-21T12:24:07.751 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-21T12:24:07.751 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-09-21T12:24:07.751 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-09-21T12:24:07.751 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout } 2024-09-21T12:24:07.751 INFO:teuthology.orchestra.run.smithi079.stdout:Waiting for the mgr to restart... 2024-09-21T12:24:07.751 INFO:teuthology.orchestra.run.smithi079.stdout:Waiting for mgr epoch 9... 2024-09-21T12:24:08.049 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:07 smithi079 ceph-mon[23699]: from='client.? 172.21.15.79:0/4239920709' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-09-21T12:24:08.442 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:24:08 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: 2024-09-21T12:24:08.141+0000 7f0e719e5040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-21T12:24:08.442 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:24:08 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: 2024-09-21T12:24:08.207+0000 7f0e719e5040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-21T12:24:09.171 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:24:08 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: 2024-09-21T12:24:08.914+0000 7f0e719e5040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-21T12:24:09.171 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:24:09 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: 2024-09-21T12:24:09.028+0000 7f0e719e5040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-21T12:24:09.442 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:24:09 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: 2024-09-21T12:24:09.168+0000 7f0e719e5040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-21T12:24:09.442 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:24:09 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: 2024-09-21T12:24:09.229+0000 7f0e719e5040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-21T12:24:09.442 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:24:09 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: 2024-09-21T12:24:09.375+0000 7f0e719e5040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-21T12:24:09.724 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:24:09 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: 2024-09-21T12:24:09.443+0000 7f0e719e5040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-21T12:24:09.724 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:24:09 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: 2024-09-21T12:24:09.502+0000 7f0e719e5040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-21T12:24:09.724 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:24:09 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: 2024-09-21T12:24:09.657+0000 7f0e719e5040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-21T12:24:09.724 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:24:09 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: 2024-09-21T12:24:09.722+0000 7f0e719e5040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-21T12:24:09.987 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:24:09 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: 2024-09-21T12:24:09.787+0000 7f0e719e5040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-21T12:24:09.987 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:24:09 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: 2024-09-21T12:24:09.869+0000 7f0e719e5040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-21T12:24:09.987 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:24:09 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: 2024-09-21T12:24:09.928+0000 7f0e719e5040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-21T12:24:09.987 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:24:09 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: 2024-09-21T12:24:09.985+0000 7f0e719e5040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-21T12:24:11.192 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:24:10 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: 2024-09-21T12:24:10.758+0000 7f0e719e5040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-21T12:24:11.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:10 smithi079 ceph-mon[23699]: Active manager daemon y restarted 2024-09-21T12:24:11.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:10 smithi079 ceph-mon[23699]: Activating manager daemon y 2024-09-21T12:24:11.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:10 smithi079 ceph-mon[23699]: osdmap e3: 0 total, 0 up, 0 in 2024-09-21T12:24:11.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:10 smithi079 ceph-mon[23699]: mgrmap e10: y(active, starting, since 0.00307248s) 2024-09-21T12:24:11.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:10 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T12:24:11.193 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:10 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-21T12:24:11.193 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:10 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-21T12:24:11.193 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:10 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-21T12:24:11.193 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:10 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-21T12:24:11.193 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:10 smithi079 ceph-mon[23699]: Manager daemon y is now available 2024-09-21T12:24:11.193 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:10 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:24:11.193 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:10 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:24:11.193 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:10 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-21T12:24:11.193 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:10 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-21T12:24:12.358 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout { 2024-09-21T12:24:12.358 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-09-21T12:24:12.358 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout "initialized": true 2024-09-21T12:24:12.358 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout } 2024-09-21T12:24:12.359 INFO:teuthology.orchestra.run.smithi079.stdout:mgr epoch 9 is available 2024-09-21T12:24:12.359 INFO:teuthology.orchestra.run.smithi079.stdout:Generating a dashboard self-signed certificate... 2024-09-21T12:24:13.011 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:12 smithi079 ceph-mon[23699]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-09-21T12:24:13.011 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:12 smithi079 ceph-mon[23699]: mgrmap e11: y(active, since 1.00571s) 2024-09-21T12:24:13.011 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:12 smithi079 ceph-mon[23699]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-09-21T12:24:13.011 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:12 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:24:13.011 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:12 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi079", "name": "osd_memory_target"}]: dispatch 2024-09-21T12:24:13.011 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:12 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:24:13.011 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:12 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi079", "caps": []}]: dispatch 2024-09-21T12:24:13.012 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:24:12 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: 2024-09-21T12:24:12.849+0000 7f0dd7e2f640 -1 log_channel(cephadm) log [ERR] : [21/Sep/2024:12:24:12] ENGINE Error in HTTPServer.serve 2024-09-21T12:24:13.012 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:24:12 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: Traceback (most recent call last): 2024-09-21T12:24:13.012 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:24:12 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T12:24:13.012 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:24:12 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: self._connections.run(self.expiration_interval) 2024-09-21T12:24:13.012 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:24:12 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T12:24:13.012 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:24:12 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: self._run(expiration_interval) 2024-09-21T12:24:13.012 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:24:12 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T12:24:13.012 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:24:12 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T12:24:13.012 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:24:12 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T12:24:13.013 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:24:12 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T12:24:13.013 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:24:12 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T12:24:13.013 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:24:12 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: s = self.context.wrap_socket( 2024-09-21T12:24:13.013 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:24:12 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T12:24:13.013 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:24:12 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: return self.sslsocket_class._create( 2024-09-21T12:24:13.013 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:24:12 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T12:24:13.013 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:24:12 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: self.do_handshake() 2024-09-21T12:24:13.013 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:24:12 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T12:24:13.013 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:24:12 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: self._sslobj.do_handshake() 2024-09-21T12:24:13.013 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:24:12 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T12:24:13.013 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:24:12 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: 2024-09-21T12:24:13.623 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-09-21T12:24:13.623 INFO:teuthology.orchestra.run.smithi079.stdout:Creating initial admin user... 2024-09-21T12:24:14.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:14 smithi079 ceph-mon[23699]: [21/Sep/2024:12:24:12] ENGINE Bus STARTING 2024-09-21T12:24:14.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:14 smithi079 ceph-mon[23699]: [21/Sep/2024:12:24:12] ENGINE Serving on http://172.21.15.79:8765 2024-09-21T12:24:14.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:14 smithi079 ceph-mon[23699]: [21/Sep/2024:12:24:12] ENGINE Serving on https://172.21.15.79:7150 2024-09-21T12:24:14.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:14 smithi079 ceph-mon[23699]: [21/Sep/2024:12:24:12] ENGINE Bus STARTED 2024-09-21T12:24:14.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:14 smithi079 ceph-mon[23699]: [21/Sep/2024:12:24:12] ENGINE Error in HTTPServer.serve 2024-09-21T12:24:14.192 INFO:journalctl@ceph.mon.a.smithi079.stdout: Traceback (most recent call last): 2024-09-21T12:24:14.192 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T12:24:14.192 INFO:journalctl@ceph.mon.a.smithi079.stdout: self._connections.run(self.expiration_interval) 2024-09-21T12:24:14.192 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T12:24:14.193 INFO:journalctl@ceph.mon.a.smithi079.stdout: self._run(expiration_interval) 2024-09-21T12:24:14.193 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T12:24:14.193 INFO:journalctl@ceph.mon.a.smithi079.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T12:24:14.193 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T12:24:14.193 INFO:journalctl@ceph.mon.a.smithi079.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T12:24:14.193 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T12:24:14.193 INFO:journalctl@ceph.mon.a.smithi079.stdout: s = self.context.wrap_socket( 2024-09-21T12:24:14.193 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T12:24:14.193 INFO:journalctl@ceph.mon.a.smithi079.stdout: return self.sslsocket_class._create( 2024-09-21T12:24:14.193 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T12:24:14.193 INFO:journalctl@ceph.mon.a.smithi079.stdout: self.do_handshake() 2024-09-21T12:24:14.193 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T12:24:14.193 INFO:journalctl@ceph.mon.a.smithi079.stdout: self._sslobj.do_handshake() 2024-09-21T12:24:14.193 INFO:journalctl@ceph.mon.a.smithi079.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T12:24:14.193 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:14 smithi079 ceph-mon[23699]: from='client.14162 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:24:14.193 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:14 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:24:14.194 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:14 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:24:14.194 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:14 smithi079 ceph-mon[23699]: mgrmap e12: y(active, since 2s) 2024-09-21T12:24:15.003 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$35adS2hhiM64nPWn1WHXReJx7jEpDTAwuG0hubsIbHk8MYU0KHTQu", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1726921454, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-09-21T12:24:15.003 INFO:teuthology.orchestra.run.smithi079.stdout:Fetching dashboard port number... 2024-09-21T12:24:15.312 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:15 smithi079 ceph-mon[23699]: Deploying daemon agent.smithi079 on smithi079 2024-09-21T12:24:15.312 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:15 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:24:16.166 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:16 smithi079 ceph-mon[23699]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:24:16.166 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:16 smithi079 ceph-mon[23699]: from='client.? 172.21.15.79:0/2408676112' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-09-21T12:24:16.168 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stdout 8443 2024-09-21T12:24:16.179 INFO:teuthology.orchestra.run.smithi079.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-21T12:24:16.179 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stdout disabled 2024-09-21T12:24:16.186 INFO:teuthology.orchestra.run.smithi079.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-21T12:24:16.186 INFO:teuthology.orchestra.run.smithi079.stdout:systemctl: stdout inactive 2024-09-21T12:24:16.186 INFO:teuthology.orchestra.run.smithi079.stdout:firewalld.service is not enabled 2024-09-21T12:24:16.186 INFO:teuthology.orchestra.run.smithi079.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-09-21T12:24:16.188 INFO:teuthology.orchestra.run.smithi079.stdout:Ceph Dashboard is now available at: 2024-09-21T12:24:16.188 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-21T12:24:16.189 INFO:teuthology.orchestra.run.smithi079.stdout: URL: https://smithi079.front.sepia.ceph.com:8443/ 2024-09-21T12:24:16.189 INFO:teuthology.orchestra.run.smithi079.stdout: User: admin 2024-09-21T12:24:16.189 INFO:teuthology.orchestra.run.smithi079.stdout: Password: aefwf13pb3 2024-09-21T12:24:16.189 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-21T12:24:16.189 INFO:teuthology.orchestra.run.smithi079.stdout:Saving cluster configuration to /var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/config directory 2024-09-21T12:24:16.189 INFO:teuthology.orchestra.run.smithi079.stdout:Enabling autotune for osd_memory_target 2024-09-21T12:24:18.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:18 smithi079 ceph-mon[23699]: mgrmap e13: y(active, since 6s) 2024-09-21T12:24:18.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:18 smithi079 ceph-mon[23699]: from='client.? 172.21.15.79:0/4114017245' entity='client.admin' 2024-09-21T12:24:18.555 INFO:teuthology.orchestra.run.smithi079.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-09-21T12:24:18.556 INFO:teuthology.orchestra.run.smithi079.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-09-21T12:24:18.556 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-21T12:24:18.556 INFO:teuthology.orchestra.run.smithi079.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 33168022-7814-11ef-baf6-efdc52797490 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-09-21T12:24:18.556 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-21T12:24:18.556 INFO:teuthology.orchestra.run.smithi079.stdout:Or, if you are only running a single cluster on this host: 2024-09-21T12:24:18.556 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-21T12:24:18.556 INFO:teuthology.orchestra.run.smithi079.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-09-21T12:24:18.556 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-21T12:24:18.556 INFO:teuthology.orchestra.run.smithi079.stdout:Please consider enabling telemetry to help improve Ceph: 2024-09-21T12:24:18.556 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-21T12:24:18.556 INFO:teuthology.orchestra.run.smithi079.stdout: ceph telemetry on 2024-09-21T12:24:18.556 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-21T12:24:18.556 INFO:teuthology.orchestra.run.smithi079.stdout:For more information see: 2024-09-21T12:24:18.556 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-21T12:24:18.557 INFO:teuthology.orchestra.run.smithi079.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-09-21T12:24:18.557 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-21T12:24:18.557 INFO:teuthology.orchestra.run.smithi079.stdout:Bootstrap complete. 2024-09-21T12:24:18.592 INFO:tasks.cephadm:Fetching config... 2024-09-21T12:24:18.592 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-09-21T12:24:18.592 DEBUG:teuthology.orchestra.run.smithi079:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-09-21T12:24:18.608 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-09-21T12:24:18.608 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-09-21T12:24:18.608 DEBUG:teuthology.orchestra.run.smithi079:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-09-21T12:24:18.671 INFO:tasks.cephadm:Fetching mon keyring... 2024-09-21T12:24:18.671 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-09-21T12:24:18.671 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd if=/var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/mon.a/keyring of=/dev/stdout 2024-09-21T12:24:18.784 INFO:tasks.cephadm:Fetching pub ssh key... 2024-09-21T12:24:18.784 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-09-21T12:24:18.784 DEBUG:teuthology.orchestra.run.smithi079:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-09-21T12:24:18.849 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-09-21T12:24:18.849 DEBUG:teuthology.orchestra.run.smithi079:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQChPe7ps6ciUCQf0G1KohygC2ZQg803O4LrtJD51v3f8R62etuXbYv7dh5O5jSYrbex4BkCxItIPTukYylXqGgLjHrhPtCIDDmLQipr929jXTMSyIpYxuzeDZuCzlyuYMRuwzV3OX48O5GWrodAQGKs/agdKncpD+fxkqTGNzy90wK1LcKkzowZHdQIhsi7oxDQQ0M+ffOQH3glGovHV118YieCBWE9vO8F3W6QSIuOr62fv3IPeXc4aEtlHtpEEN3LOJei5QuV+MoRBjJgrV/NC8NHe5YkCO8Mk5x3dFGZiwOGM/qoHAlrJYpLdDpzCpijSHtsfMPpKatWeX1oLhhgdDeSwQsdY6KRCd+bIrHZ38bcJviuIh3k6i49OEsQ9csmXWIiZQDVpEGXBSzUPIUIcQIKf3bede2yme1DN1n1r9bVqtoY3Yf4W1cdwE7kIzsJjhJFO7hSDNK2Tqx5+fgXnPAiNA+CI/ZU27jx2k0tMoSxl0uB+UmOv5/r/463I+0= ceph-33168022-7814-11ef-baf6-efdc52797490' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-09-21T12:24:18.983 INFO:teuthology.orchestra.run.smithi079.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQChPe7ps6ciUCQf0G1KohygC2ZQg803O4LrtJD51v3f8R62etuXbYv7dh5O5jSYrbex4BkCxItIPTukYylXqGgLjHrhPtCIDDmLQipr929jXTMSyIpYxuzeDZuCzlyuYMRuwzV3OX48O5GWrodAQGKs/agdKncpD+fxkqTGNzy90wK1LcKkzowZHdQIhsi7oxDQQ0M+ffOQH3glGovHV118YieCBWE9vO8F3W6QSIuOr62fv3IPeXc4aEtlHtpEEN3LOJei5QuV+MoRBjJgrV/NC8NHe5YkCO8Mk5x3dFGZiwOGM/qoHAlrJYpLdDpzCpijSHtsfMPpKatWeX1oLhhgdDeSwQsdY6KRCd+bIrHZ38bcJviuIh3k6i49OEsQ9csmXWIiZQDVpEGXBSzUPIUIcQIKf3bede2yme1DN1n1r9bVqtoY3Yf4W1cdwE7kIzsJjhJFO7hSDNK2Tqx5+fgXnPAiNA+CI/ZU27jx2k0tMoSxl0uB+UmOv5/r/463I+0= ceph-33168022-7814-11ef-baf6-efdc52797490 2024-09-21T12:24:18.997 DEBUG:teuthology.orchestra.run.smithi104:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQChPe7ps6ciUCQf0G1KohygC2ZQg803O4LrtJD51v3f8R62etuXbYv7dh5O5jSYrbex4BkCxItIPTukYylXqGgLjHrhPtCIDDmLQipr929jXTMSyIpYxuzeDZuCzlyuYMRuwzV3OX48O5GWrodAQGKs/agdKncpD+fxkqTGNzy90wK1LcKkzowZHdQIhsi7oxDQQ0M+ffOQH3glGovHV118YieCBWE9vO8F3W6QSIuOr62fv3IPeXc4aEtlHtpEEN3LOJei5QuV+MoRBjJgrV/NC8NHe5YkCO8Mk5x3dFGZiwOGM/qoHAlrJYpLdDpzCpijSHtsfMPpKatWeX1oLhhgdDeSwQsdY6KRCd+bIrHZ38bcJviuIh3k6i49OEsQ9csmXWIiZQDVpEGXBSzUPIUIcQIKf3bede2yme1DN1n1r9bVqtoY3Yf4W1cdwE7kIzsJjhJFO7hSDNK2Tqx5+fgXnPAiNA+CI/ZU27jx2k0tMoSxl0uB+UmOv5/r/463I+0= ceph-33168022-7814-11ef-baf6-efdc52797490' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-09-21T12:24:19.055 INFO:teuthology.orchestra.run.smithi104.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQChPe7ps6ciUCQf0G1KohygC2ZQg803O4LrtJD51v3f8R62etuXbYv7dh5O5jSYrbex4BkCxItIPTukYylXqGgLjHrhPtCIDDmLQipr929jXTMSyIpYxuzeDZuCzlyuYMRuwzV3OX48O5GWrodAQGKs/agdKncpD+fxkqTGNzy90wK1LcKkzowZHdQIhsi7oxDQQ0M+ffOQH3glGovHV118YieCBWE9vO8F3W6QSIuOr62fv3IPeXc4aEtlHtpEEN3LOJei5QuV+MoRBjJgrV/NC8NHe5YkCO8Mk5x3dFGZiwOGM/qoHAlrJYpLdDpzCpijSHtsfMPpKatWeX1oLhhgdDeSwQsdY6KRCd+bIrHZ38bcJviuIh3k6i49OEsQ9csmXWIiZQDVpEGXBSzUPIUIcQIKf3bede2yme1DN1n1r9bVqtoY3Yf4W1cdwE7kIzsJjhJFO7hSDNK2Tqx5+fgXnPAiNA+CI/ZU27jx2k0tMoSxl0uB+UmOv5/r/463I+0= ceph-33168022-7814-11ef-baf6-efdc52797490 2024-09-21T12:24:19.068 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 33168022-7814-11ef-baf6-efdc52797490 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-09-21T12:24:19.446 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:24:20.604 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:20 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:24:20.604 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:20 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:24:20.604 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:20 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:24:20.604 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:20 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:24:20.605 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:20 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:24:20.605 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:20 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:24:20.605 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:20 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:24:20.605 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:20 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:24:20.605 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:20 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:24:20.605 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:20 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:24:20.605 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:20 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:24:20.605 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:20 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:24:21.798 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:21 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:24:21.798 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:21 smithi079 ceph-mon[23699]: from='client.? 172.21.15.79:0/1384126018' entity='client.admin' 2024-09-21T12:24:21.798 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:21 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:24:21.798 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:21 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:24:23.340 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-09-21T12:24:23.340 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 33168022-7814-11ef-baf6-efdc52797490 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-09-21T12:24:23.854 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:24:24.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:24 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:24:24.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:24 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:24:26.698 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi104 2024-09-21T12:24:26.698 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-09-21T12:24:26.698 DEBUG:teuthology.orchestra.run.smithi104:> dd of=/etc/ceph/ceph.conf 2024-09-21T12:24:26.717 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-09-21T12:24:26.717 DEBUG:teuthology.orchestra.run.smithi104:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-21T12:24:26.775 INFO:tasks.cephadm:Adding host smithi104 to orchestrator... 2024-09-21T12:24:26.775 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 33168022-7814-11ef-baf6-efdc52797490 -- ceph orch host add smithi104 2024-09-21T12:24:26.797 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:26 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:24:26.797 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:26 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:24:26.797 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:26 smithi079 ceph-mon[23699]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:24:26.797 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:26 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:24:26.797 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:26 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:24:26.797 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:26 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:24:26.797 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:26 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:24:26.797 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:26 smithi079 ceph-mon[23699]: Updating smithi079:/etc/ceph/ceph.conf 2024-09-21T12:24:26.966 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:24:27.973 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:27 smithi079 ceph-mon[23699]: Updating smithi079:/var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T12:24:27.973 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:27 smithi079 ceph-mon[23699]: Updating smithi079:/etc/ceph/ceph.client.admin.keyring 2024-09-21T12:24:27.973 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:27 smithi079 ceph-mon[23699]: Updating smithi079:/var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T12:24:27.973 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:27 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:24:27.973 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:27 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:24:27.973 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:27 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:24:29.441 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:29 smithi079 ceph-mon[23699]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi104", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:24:30.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:30 smithi079 ceph-mon[23699]: Deploying cephadm binary to smithi104 2024-09-21T12:24:31.245 INFO:teuthology.orchestra.run.smithi079.stdout:Added host 'smithi104' with addr '172.21.15.104' 2024-09-21T12:24:31.362 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:31 smithi079 ceph-mon[23699]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:24:31.800 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 33168022-7814-11ef-baf6-efdc52797490 -- ceph orch host ls --format=json 2024-09-21T12:24:32.032 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:24:32.404 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:32 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:24:32.404 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:32 smithi079 ceph-mon[23699]: Added host smithi104 2024-09-21T12:24:32.404 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:32 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:24:32.404 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:32 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:24:32.404 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:32 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:24:32.404 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:32 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:24:32.404 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:32 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi104", "name": "osd_memory_target"}]: dispatch 2024-09-21T12:24:32.404 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:32 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:24:32.404 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:32 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:24:32.404 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:32 smithi079 ceph-mon[23699]: Updating smithi104:/etc/ceph/ceph.conf 2024-09-21T12:24:32.404 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:32 smithi079 ceph-mon[23699]: Updating smithi104:/var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T12:24:33.585 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-21T12:24:33.586 INFO:teuthology.orchestra.run.smithi079.stdout:[{"addr": "172.21.15.79", "hostname": "smithi079", "labels": [], "status": ""}, {"addr": "172.21.15.104", "hostname": "smithi104", "labels": [], "status": ""}] 2024-09-21T12:24:33.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:33 smithi079 ceph-mon[23699]: Updating smithi104:/etc/ceph/ceph.client.admin.keyring 2024-09-21T12:24:33.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:33 smithi079 ceph-mon[23699]: Updating smithi104:/var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T12:24:33.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:33 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:24:33.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:33 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:24:33.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:33 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi104", "caps": []}]: dispatch 2024-09-21T12:24:33.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:33 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.agent.smithi104", "caps": []}]': finished 2024-09-21T12:24:33.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:33 smithi079 ceph-mon[23699]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:24:34.229 INFO:tasks.cephadm:Setting crush tunables to default 2024-09-21T12:24:34.229 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 33168022-7814-11ef-baf6-efdc52797490 -- ceph osd crush tunables default 2024-09-21T12:24:34.415 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:24:35.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:34 smithi079 ceph-mon[23699]: Deploying daemon agent.smithi104 on smithi104 2024-09-21T12:24:35.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:34 smithi079 ceph-mon[23699]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T12:24:35.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:34 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:24:35.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:34 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:24:35.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:34 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:24:35.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:34 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:24:35.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:34 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:24:35.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:34 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:24:35.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:34 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:24:35.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:34 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:24:35.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:34 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:24:35.193 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:34 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:24:35.193 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:34 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:24:35.193 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:34 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:24:35.193 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:34 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:24:35.193 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:34 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:24:36.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:35 smithi079 ceph-mon[23699]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:24:36.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:35 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:24:36.950 INFO:teuthology.orchestra.run.smithi079.stderr:adjusted tunables profile to default 2024-09-21T12:24:37.254 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:36 smithi079 ceph-mon[23699]: from='client.? 172.21.15.79:0/3618158966' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-09-21T12:24:37.546 INFO:tasks.cephadm:Adding mon.a on smithi079 2024-09-21T12:24:37.547 INFO:tasks.cephadm:Adding mon.c on smithi079 2024-09-21T12:24:37.547 INFO:tasks.cephadm:Adding mon.b on smithi104 2024-09-21T12:24:37.547 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 33168022-7814-11ef-baf6-efdc52797490 -- ceph orch apply mon '3;smithi079:172.21.15.79=a;smithi079:[v2:172.21.15.79:3301,v1:172.21.15.79:6790]=c;smithi104:172.21.15.104=b' 2024-09-21T12:24:37.738 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T12:24:38.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:37 smithi079 ceph-mon[23699]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:24:38.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:37 smithi079 ceph-mon[23699]: from='client.? 172.21.15.79:0/3618158966' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-09-21T12:24:38.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:37 smithi079 ceph-mon[23699]: osdmap e4: 0 total, 0 up, 0 in 2024-09-21T12:24:38.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:37 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:24:38.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:37 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:24:39.272 INFO:teuthology.orchestra.run.smithi104.stdout:Scheduled mon update... 2024-09-21T12:24:39.822 DEBUG:teuthology.orchestra.run.smithi079:mon.c> sudo journalctl -f -n 0 -u ceph-33168022-7814-11ef-baf6-efdc52797490@mon.c.service 2024-09-21T12:24:39.825 DEBUG:teuthology.orchestra.run.smithi104:mon.b> sudo journalctl -f -n 0 -u ceph-33168022-7814-11ef-baf6-efdc52797490@mon.b.service 2024-09-21T12:24:39.827 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-21T12:24:39.827 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 33168022-7814-11ef-baf6-efdc52797490 -- ceph mon dump -f json 2024-09-21T12:24:40.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:39 smithi079 ceph-mon[23699]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:24:40.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:39 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:24:40.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:39 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:24:40.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:39 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:24:40.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:39 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:24:40.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:39 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:24:40.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:39 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-21T12:24:40.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:39 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:24:40.485 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T12:24:41.049 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:40 smithi079 ceph-mon[23699]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi079:172.21.15.79=a;smithi079:[v2:172.21.15.79:3301,v1:172.21.15.79:6790]=c;smithi104:172.21.15.104=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:24:41.049 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:40 smithi079 ceph-mon[23699]: Saving service mon spec with placement smithi079:172.21.15.79=a;smithi079:[v2:172.21.15.79:3301,v1:172.21.15.79:6790]=c;smithi104:172.21.15.104=b;count:3 2024-09-21T12:24:41.050 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:40 smithi079 ceph-mon[23699]: Deploying daemon mon.b on smithi104 2024-09-21T12:24:42.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:41 smithi079 ceph-mon[23699]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:24:42.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:41 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:24:42.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:41 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:24:42.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:41 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:24:42.979 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-21T12:24:42.979 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":1,"fsid":"33168022-7814-11ef-baf6-efdc52797490","modified":"2024-09-21T12:23:19.108132Z","created":"2024-09-21T12:23:19.108132Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:3300","nonce":0},{"type":"v1","addr":"172.21.15.79:6789","nonce":0}]},"addr":"172.21.15.79:6789/0","public_addr":"172.21.15.79:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-21T12:24:42.980 INFO:teuthology.orchestra.run.smithi104.stderr:dumped monmap epoch 1 2024-09-21T12:24:44.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 podman[25620]: 2024-09-21 12:24:44.293852509 +0000 UTC m=+0.015642581 image pull 0adab276c97e6f3143ffbca5eeff37fcd7dc3b5c0f339a9f747381d9993602de quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6 2024-09-21T12:24:44.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:43 smithi079 ceph-mon[23699]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:24:44.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:43 smithi079 ceph-mon[23699]: from='client.? 172.21.15.104:0/4019363617' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-21T12:24:44.882 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-21T12:24:44.882 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 33168022-7814-11ef-baf6-efdc52797490 -- ceph mon dump -f json 2024-09-21T12:24:44.905 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 podman[25620]: 2024-09-21 12:24:44.604159666 +0000 UTC m=+0.325949740 container init 460a6e8db654c089a736529f410a34461581257919fd6bf9bc711f1bdb5830fe (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-33168022-7814-11ef-baf6-efdc52797490-mon-b, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=reef-6a1bdf8, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-09-21T12:24:44.906 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 podman[25620]: 2024-09-21 12:24:44.607178475 +0000 UTC m=+0.328968542 container start 460a6e8db654c089a736529f410a34461581257919fd6bf9bc711f1bdb5830fe (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-33168022-7814-11ef-baf6-efdc52797490-mon-b, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=reef-6a1bdf8, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-09-21T12:24:44.906 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: set uid:gid to 167:167 (ceph:ceph) 2024-09-21T12:24:44.906 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: ceph version 18.2.4-871-g6a1bdf85 (6a1bdf854bb5803efd58957907442aa01f41a8fc) reef (stable), process ceph-mon, pid 2 2024-09-21T12:24:44.906 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: pidfile_write: ignore empty --pid-file 2024-09-21T12:24:44.906 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: load: jerasure load: lrc 2024-09-21T12:24:44.906 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: RocksDB version: 7.9.2 2024-09-21T12:24:44.906 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Git sha 0 2024-09-21T12:24:44.906 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Compile date 2024-09-20 20:34:00 2024-09-21T12:24:44.906 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: DB SUMMARY 2024-09-21T12:24:44.906 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: DB Session ID: ZMMFHQM86889W4N0HLUZ 2024-09-21T12:24:44.906 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: CURRENT file: CURRENT 2024-09-21T12:24:44.906 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: IDENTITY file: IDENTITY 2024-09-21T12:24:44.906 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-09-21T12:24:44.907 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2024-09-21T12:24:44.907 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 511 ; 2024-09-21T12:24:44.907 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.error_if_exists: 0 2024-09-21T12:24:44.907 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.create_if_missing: 0 2024-09-21T12:24:44.907 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.paranoid_checks: 1 2024-09-21T12:24:44.907 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.flush_verify_memtable_count: 1 2024-09-21T12:24:44.907 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-21T12:24:44.907 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-09-21T12:24:44.907 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.env: 0x55a9aa045bc0 2024-09-21T12:24:44.907 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.fs: PosixFileSystem 2024-09-21T12:24:44.907 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.info_log: 0x55a9aabeebe0 2024-09-21T12:24:44.907 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.max_file_opening_threads: 16 2024-09-21T12:24:44.907 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.statistics: (nil) 2024-09-21T12:24:44.909 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.use_fsync: 0 2024-09-21T12:24:44.909 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.max_log_file_size: 0 2024-09-21T12:24:44.909 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-21T12:24:44.909 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-21T12:24:44.909 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.keep_log_file_num: 1000 2024-09-21T12:24:44.909 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.recycle_log_file_num: 0 2024-09-21T12:24:44.909 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.allow_fallocate: 1 2024-09-21T12:24:44.909 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.allow_mmap_reads: 0 2024-09-21T12:24:44.909 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.allow_mmap_writes: 0 2024-09-21T12:24:44.910 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.use_direct_reads: 0 2024-09-21T12:24:44.910 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-21T12:24:44.910 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.create_missing_column_families: 0 2024-09-21T12:24:44.910 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.db_log_dir: 2024-09-21T12:24:44.910 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.wal_dir: 2024-09-21T12:24:44.910 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-21T12:24:44.910 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-21T12:24:44.910 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-21T12:24:44.910 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-21T12:24:44.910 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-21T12:24:44.910 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-21T12:24:44.910 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.advise_random_on_open: 1 2024-09-21T12:24:44.910 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.db_write_buffer_size: 0 2024-09-21T12:24:44.911 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.write_buffer_manager: 0x55a9aabfc320 2024-09-21T12:24:44.911 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-21T12:24:44.911 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-21T12:24:44.911 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-21T12:24:44.911 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.rate_limiter: (nil) 2024-09-21T12:24:44.911 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-21T12:24:44.911 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.wal_recovery_mode: 2 2024-09-21T12:24:44.911 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.enable_thread_tracking: 0 2024-09-21T12:24:44.911 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.enable_pipelined_write: 0 2024-09-21T12:24:44.911 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.unordered_write: 0 2024-09-21T12:24:44.911 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-21T12:24:44.911 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-21T12:24:44.911 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-21T12:24:44.911 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-21T12:24:44.911 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.row_cache: None 2024-09-21T12:24:44.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.wal_filter: None 2024-09-21T12:24:44.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-21T12:24:44.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.allow_ingest_behind: 0 2024-09-21T12:24:44.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.two_write_queues: 0 2024-09-21T12:24:44.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.manual_wal_flush: 0 2024-09-21T12:24:44.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.wal_compression: 0 2024-09-21T12:24:44.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.atomic_flush: 0 2024-09-21T12:24:44.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-21T12:24:44.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-21T12:24:44.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-21T12:24:44.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.log_readahead_size: 0 2024-09-21T12:24:44.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-21T12:24:44.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.best_efforts_recovery: 0 2024-09-21T12:24:44.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-21T12:24:44.912 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-21T12:24:44.913 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.allow_data_in_errors: 0 2024-09-21T12:24:44.913 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.db_host_id: __hostname__ 2024-09-21T12:24:44.913 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.enforce_single_del_contracts: true 2024-09-21T12:24:44.913 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.max_background_jobs: 2 2024-09-21T12:24:44.913 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.max_background_compactions: -1 2024-09-21T12:24:44.913 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.max_subcompactions: 1 2024-09-21T12:24:44.913 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-21T12:24:44.913 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-21T12:24:44.913 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-21T12:24:44.913 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.max_total_wal_size: 0 2024-09-21T12:24:44.913 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-21T12:24:44.913 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-21T12:24:44.913 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-21T12:24:44.913 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-21T12:24:44.913 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.max_open_files: -1 2024-09-21T12:24:44.914 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.bytes_per_sync: 0 2024-09-21T12:24:44.914 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-21T12:24:44.914 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-21T12:24:44.914 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.compaction_readahead_size: 0 2024-09-21T12:24:44.914 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.max_background_flushes: -1 2024-09-21T12:24:44.914 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Compression algorithms supported: 2024-09-21T12:24:44.914 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: kZSTD supported: 0 2024-09-21T12:24:44.914 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: kXpressCompression supported: 0 2024-09-21T12:24:44.914 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: kBZip2Compression supported: 0 2024-09-21T12:24:44.914 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-21T12:24:44.914 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: kLZ4Compression supported: 1 2024-09-21T12:24:44.914 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: kZlibCompression supported: 1 2024-09-21T12:24:44.914 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: kLZ4HCCompression supported: 1 2024-09-21T12:24:44.914 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: kSnappyCompression supported: 1 2024-09-21T12:24:44.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-21T12:24:44.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: DMutex implementation: pthread_mutex_t 2024-09-21T12:24:44.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2024-09-21T12:24:44.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-09-21T12:24:44.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-21T12:24:44.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.merge_operator: 2024-09-21T12:24:44.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.compaction_filter: None 2024-09-21T12:24:44.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.compaction_filter_factory: None 2024-09-21T12:24:44.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.sst_partitioner_factory: None 2024-09-21T12:24:44.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-21T12:24:44.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-21T12:24:44.915 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55a9aabee7e0) 2024-09-21T12:24:44.915 INFO:journalctl@ceph.mon.b.smithi104.stdout: cache_index_and_filter_blocks: 1 2024-09-21T12:24:44.915 INFO:journalctl@ceph.mon.b.smithi104.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-21T12:24:44.916 INFO:journalctl@ceph.mon.b.smithi104.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-21T12:24:44.916 INFO:journalctl@ceph.mon.b.smithi104.stdout: pin_top_level_index_and_filter: 1 2024-09-21T12:24:44.916 INFO:journalctl@ceph.mon.b.smithi104.stdout: index_type: 0 2024-09-21T12:24:44.916 INFO:journalctl@ceph.mon.b.smithi104.stdout: data_block_index_type: 0 2024-09-21T12:24:44.916 INFO:journalctl@ceph.mon.b.smithi104.stdout: index_shortening: 1 2024-09-21T12:24:44.916 INFO:journalctl@ceph.mon.b.smithi104.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-21T12:24:44.916 INFO:journalctl@ceph.mon.b.smithi104.stdout: checksum: 4 2024-09-21T12:24:44.916 INFO:journalctl@ceph.mon.b.smithi104.stdout: no_block_cache: 0 2024-09-21T12:24:44.916 INFO:journalctl@ceph.mon.b.smithi104.stdout: block_cache: 0x55a9aabe3350 2024-09-21T12:24:44.916 INFO:journalctl@ceph.mon.b.smithi104.stdout: block_cache_name: BinnedLRUCache 2024-09-21T12:24:44.916 INFO:journalctl@ceph.mon.b.smithi104.stdout: block_cache_options: 2024-09-21T12:24:44.916 INFO:journalctl@ceph.mon.b.smithi104.stdout: capacity : 536870912 2024-09-21T12:24:44.916 INFO:journalctl@ceph.mon.b.smithi104.stdout: num_shard_bits : 4 2024-09-21T12:24:44.916 INFO:journalctl@ceph.mon.b.smithi104.stdout: strict_capacity_limit : 0 2024-09-21T12:24:44.917 INFO:journalctl@ceph.mon.b.smithi104.stdout: high_pri_pool_ratio: 0.000 2024-09-21T12:24:44.917 INFO:journalctl@ceph.mon.b.smithi104.stdout: block_cache_compressed: (nil) 2024-09-21T12:24:44.917 INFO:journalctl@ceph.mon.b.smithi104.stdout: persistent_cache: (nil) 2024-09-21T12:24:44.917 INFO:journalctl@ceph.mon.b.smithi104.stdout: block_size: 4096 2024-09-21T12:24:44.917 INFO:journalctl@ceph.mon.b.smithi104.stdout: block_size_deviation: 10 2024-09-21T12:24:44.917 INFO:journalctl@ceph.mon.b.smithi104.stdout: block_restart_interval: 16 2024-09-21T12:24:44.917 INFO:journalctl@ceph.mon.b.smithi104.stdout: index_block_restart_interval: 1 2024-09-21T12:24:44.917 INFO:journalctl@ceph.mon.b.smithi104.stdout: metadata_block_size: 4096 2024-09-21T12:24:44.917 INFO:journalctl@ceph.mon.b.smithi104.stdout: partition_filters: 0 2024-09-21T12:24:44.917 INFO:journalctl@ceph.mon.b.smithi104.stdout: use_delta_encoding: 1 2024-09-21T12:24:44.917 INFO:journalctl@ceph.mon.b.smithi104.stdout: filter_policy: bloomfilter 2024-09-21T12:24:44.917 INFO:journalctl@ceph.mon.b.smithi104.stdout: whole_key_filtering: 1 2024-09-21T12:24:44.917 INFO:journalctl@ceph.mon.b.smithi104.stdout: verify_compression: 0 2024-09-21T12:24:44.917 INFO:journalctl@ceph.mon.b.smithi104.stdout: read_amp_bytes_per_bit: 0 2024-09-21T12:24:44.917 INFO:journalctl@ceph.mon.b.smithi104.stdout: format_version: 5 2024-09-21T12:24:44.917 INFO:journalctl@ceph.mon.b.smithi104.stdout: enable_index_compression: 1 2024-09-21T12:24:44.918 INFO:journalctl@ceph.mon.b.smithi104.stdout: block_align: 0 2024-09-21T12:24:44.918 INFO:journalctl@ceph.mon.b.smithi104.stdout: max_auto_readahead_size: 262144 2024-09-21T12:24:44.918 INFO:journalctl@ceph.mon.b.smithi104.stdout: prepopulate_block_cache: 0 2024-09-21T12:24:44.918 INFO:journalctl@ceph.mon.b.smithi104.stdout: initial_auto_readahead_size: 8192 2024-09-21T12:24:44.918 INFO:journalctl@ceph.mon.b.smithi104.stdout: num_file_reads_for_auto_readahead: 2 2024-09-21T12:24:44.918 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.write_buffer_size: 33554432 2024-09-21T12:24:44.918 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.max_write_buffer_number: 2 2024-09-21T12:24:44.918 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.compression: NoCompression 2024-09-21T12:24:44.918 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.bottommost_compression: Disabled 2024-09-21T12:24:44.918 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.prefix_extractor: nullptr 2024-09-21T12:24:44.918 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-21T12:24:44.918 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.num_levels: 7 2024-09-21T12:24:44.918 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-21T12:24:44.918 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-21T12:24:44.918 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-21T12:24:44.919 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-21T12:24:44.919 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-21T12:24:44.919 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-21T12:24:44.919 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-21T12:24:44.919 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-21T12:24:44.919 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-21T12:24:44.919 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-21T12:24:44.919 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-09-21T12:24:44.919 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-09-21T12:24:44.919 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-21T12:24:44.919 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.compression_opts.level: 32767 2024-09-21T12:24:44.919 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.compression_opts.strategy: 0 2024-09-21T12:24:44.919 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-21T12:24:44.919 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-21T12:24:44.919 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-09-21T12:24:44.920 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-21T12:24:44.920 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.compression_opts.enabled: false 2024-09-21T12:24:44.920 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-09-21T12:24:44.920 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-21T12:24:44.920 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-21T12:24:44.920 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-21T12:24:44.920 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.target_file_size_base: 67108864 2024-09-21T12:24:44.920 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-21T12:24:44.920 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-21T12:24:44.920 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-21T12:24:44.920 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-21T12:24:44.920 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-21T12:24:44.920 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-21T12:24:44.920 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-21T12:24:44.920 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-21T12:24:44.921 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-21T12:24:44.921 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-21T12:24:44.921 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-21T12:24:44.921 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-21T12:24:44.921 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-21T12:24:44.921 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-09-21T12:24:44.921 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.arena_block_size: 1048576 2024-09-21T12:24:44.921 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-21T12:24:44.921 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-21T12:24:44.921 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.disable_auto_compactions: 0 2024-09-21T12:24:44.921 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-21T12:24:44.921 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-21T12:24:44.921 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-21T12:24:44.921 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-21T12:24:44.921 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-21T12:24:44.922 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-21T12:24:44.922 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-21T12:24:44.922 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-21T12:24:44.922 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-21T12:24:44.922 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-21T12:24:44.922 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-21T12:24:44.922 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.inplace_update_support: 0 2024-09-21T12:24:44.922 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-21T12:24:44.922 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-21T12:24:44.922 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-21T12:24:44.922 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-21T12:24:44.922 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.bloom_locality: 0 2024-09-21T12:24:44.922 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.max_successive_merges: 0 2024-09-21T12:24:44.922 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-21T12:24:44.923 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.paranoid_file_checks: 0 2024-09-21T12:24:44.923 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.force_consistency_checks: 1 2024-09-21T12:24:44.923 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.report_bg_io_stats: 0 2024-09-21T12:24:44.923 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.ttl: 2592000 2024-09-21T12:24:44.923 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-21T12:24:44.923 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-09-21T12:24:44.923 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-09-21T12:24:44.923 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.enable_blob_files: false 2024-09-21T12:24:44.923 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.min_blob_size: 0 2024-09-21T12:24:44.923 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.blob_file_size: 268435456 2024-09-21T12:24:44.923 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-21T12:24:44.923 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-21T12:24:44.923 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-21T12:24:44.923 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-09-21T12:24:44.924 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-09-21T12:24:44.924 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.blob_file_starting_level: 0 2024-09-21T12:24:44.924 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-09-21T12:24:44.924 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-09-21T12:24:44.924 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-09-21T12:24:44.924 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 0df81e32-3c15-4488-9730-30ad5357229e 2024-09-21T12:24:44.924 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726921484633900, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-09-21T12:24:44.924 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-09-21T12:24:44.924 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726921484634346, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1648, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1726921484, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "0df81e32-3c15-4488-9730-30ad5357229e", "db_session_id": "ZMMFHQM86889W4N0HLUZ", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-09-21T12:24:44.924 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726921484634440, "job": 1, "event": "recovery_finished"} 2024-09-21T12:24:44.924 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-09-21T12:24:44.924 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-21T12:24:44.924 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55a9aac0ee00 2024-09-21T12:24:44.924 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: DB pointer 0x55a9aad02000 2024-09-21T12:24:44.924 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-09-21T12:24:44.925 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-09-21T12:24:44.925 INFO:journalctl@ceph.mon.b.smithi104.stdout: ** DB Stats ** 2024-09-21T12:24:44.925 INFO:journalctl@ceph.mon.b.smithi104.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-21T12:24:44.925 INFO:journalctl@ceph.mon.b.smithi104.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-21T12:24:44.925 INFO:journalctl@ceph.mon.b.smithi104.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-21T12:24:44.925 INFO:journalctl@ceph.mon.b.smithi104.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-21T12:24:44.925 INFO:journalctl@ceph.mon.b.smithi104.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-21T12:24:44.925 INFO:journalctl@ceph.mon.b.smithi104.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-21T12:24:44.925 INFO:journalctl@ceph.mon.b.smithi104.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-21T12:24:44.925 INFO:journalctl@ceph.mon.b.smithi104.stdout: 2024-09-21T12:24:44.925 INFO:journalctl@ceph.mon.b.smithi104.stdout: ** Compaction Stats [default] ** 2024-09-21T12:24:44.925 INFO:journalctl@ceph.mon.b.smithi104.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-21T12:24:44.925 INFO:journalctl@ceph.mon.b.smithi104.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-09-21T12:24:44.925 INFO:journalctl@ceph.mon.b.smithi104.stdout: L0 1/0 1.61 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.6 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-21T12:24:44.925 INFO:journalctl@ceph.mon.b.smithi104.stdout: Sum 1/0 1.61 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.6 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-21T12:24:44.925 INFO:journalctl@ceph.mon.b.smithi104.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.6 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-21T12:24:44.925 INFO:journalctl@ceph.mon.b.smithi104.stdout: 2024-09-21T12:24:44.926 INFO:journalctl@ceph.mon.b.smithi104.stdout: ** Compaction Stats [default] ** 2024-09-21T12:24:44.926 INFO:journalctl@ceph.mon.b.smithi104.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-21T12:24:44.926 INFO:journalctl@ceph.mon.b.smithi104.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-21T12:24:44.926 INFO:journalctl@ceph.mon.b.smithi104.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.6 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-21T12:24:44.926 INFO:journalctl@ceph.mon.b.smithi104.stdout: 2024-09-21T12:24:44.926 INFO:journalctl@ceph.mon.b.smithi104.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-09-21T12:24:44.926 INFO:journalctl@ceph.mon.b.smithi104.stdout: 2024-09-21T12:24:44.926 INFO:journalctl@ceph.mon.b.smithi104.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-21T12:24:44.926 INFO:journalctl@ceph.mon.b.smithi104.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-21T12:24:44.926 INFO:journalctl@ceph.mon.b.smithi104.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-21T12:24:44.926 INFO:journalctl@ceph.mon.b.smithi104.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-21T12:24:44.926 INFO:journalctl@ceph.mon.b.smithi104.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-21T12:24:44.926 INFO:journalctl@ceph.mon.b.smithi104.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-21T12:24:44.926 INFO:journalctl@ceph.mon.b.smithi104.stdout: Cumulative compaction: 0.00 GB write, 0.60 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-21T12:24:44.926 INFO:journalctl@ceph.mon.b.smithi104.stdout: Interval compaction: 0.00 GB write, 0.60 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-21T12:24:44.927 INFO:journalctl@ceph.mon.b.smithi104.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-21T12:24:44.927 INFO:journalctl@ceph.mon.b.smithi104.stdout: Block cache BinnedLRUCache@0x55a9aabe3350#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.1e-05 secs_since: 0 2024-09-21T12:24:44.927 INFO:journalctl@ceph.mon.b.smithi104.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-09-21T12:24:44.927 INFO:journalctl@ceph.mon.b.smithi104.stdout: 2024-09-21T12:24:44.927 INFO:journalctl@ceph.mon.b.smithi104.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-21T12:24:44.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: mon.b does not exist in monmap, will attempt to join an existing cluster 2024-09-21T12:24:44.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: using public_addr v2:172.21.15.104:0/0 -> [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] 2024-09-21T12:24:44.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: starting mon.b rank -1 at public addrs [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] at bind addrs [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid 33168022-7814-11ef-baf6-efdc52797490 2024-09-21T12:24:44.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: mon.b@-1(???) e0 preinit fsid 33168022-7814-11ef-baf6-efdc52797490 2024-09-21T12:24:44.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: mon.b@-1(synchronizing).mds e1 new map 2024-09-21T12:24:44.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: mon.b@-1(synchronizing).mds e1 print_map 2024-09-21T12:24:44.927 INFO:journalctl@ceph.mon.b.smithi104.stdout: e1 2024-09-21T12:24:44.927 INFO:journalctl@ceph.mon.b.smithi104.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-21T12:24:44.927 INFO:journalctl@ceph.mon.b.smithi104.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-09-21T12:24:44.927 INFO:journalctl@ceph.mon.b.smithi104.stdout: legacy client fscid: -1 2024-09-21T12:24:44.927 INFO:journalctl@ceph.mon.b.smithi104.stdout: 2024-09-21T12:24:44.928 INFO:journalctl@ceph.mon.b.smithi104.stdout: No filesystems configured 2024-09-21T12:24:44.928 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-09-21T12:24:44.928 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-09-21T12:24:44.928 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-09-21T12:24:44.928 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-09-21T12:24:44.928 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-09-21T12:24:44.928 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-09-21T12:24:44.928 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-09-21T12:24:44.928 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-21T12:24:44.928 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-21T12:24:44.928 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-21T12:24:44.928 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: Deploying daemon agent.smithi104 on smithi104 2024-09-21T12:24:44.928 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T12:24:44.928 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:24:44.928 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:24:44.929 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:24:44.929 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:24:44.929 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:24:44.929 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:24:44.929 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:24:44.929 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:24:44.929 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:24:44.929 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:24:44.929 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:24:44.929 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:24:44.929 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:24:44.929 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:24:44.929 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:24:44.929 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:24:44.929 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: from='client.? 172.21.15.79:0/3618158966' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-09-21T12:24:44.930 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:24:44.930 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: from='client.? 172.21.15.79:0/3618158966' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-09-21T12:24:44.930 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: osdmap e4: 0 total, 0 up, 0 in 2024-09-21T12:24:44.930 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:24:44.930 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:24:44.930 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:24:44.930 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:24:44.930 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:24:44.930 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:24:44.930 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:24:44.930 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:24:44.930 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-21T12:24:44.930 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:24:44.930 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi079:172.21.15.79=a;smithi079:[v2:172.21.15.79:3301,v1:172.21.15.79:6790]=c;smithi104:172.21.15.104=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:24:44.930 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: Saving service mon spec with placement smithi079:172.21.15.79=a;smithi079:[v2:172.21.15.79:3301,v1:172.21.15.79:6790]=c;smithi104:172.21.15.104=b;count:3 2024-09-21T12:24:44.931 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: Deploying daemon mon.b on smithi104 2024-09-21T12:24:44.931 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:24:44.931 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:24:44.931 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:24:44.931 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:24:44.931 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:24:44.931 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: from='client.? 172.21.15.104:0/4019363617' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-21T12:24:44.931 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 ceph-mon[25639]: mon.b@-1(synchronizing).paxosservice(auth 1..5) refresh upgraded, format 0 -> 3 2024-09-21T12:24:44.931 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 bash[25620]: 460a6e8db654c089a736529f410a34461581257919fd6bf9bc711f1bdb5830fe 2024-09-21T12:24:44.931 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:44 smithi104 systemd[1]: Started Ceph mon.b for 33168022-7814-11ef-baf6-efdc52797490. 2024-09-21T12:24:45.085 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:24:49.974 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:49 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T12:24:49.974 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:49 smithi104 ceph-mon[25639]: mon.a calling monitor election 2024-09-21T12:24:49.974 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:49 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T12:24:49.974 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:49 smithi104 ceph-mon[25639]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:24:49.974 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:49 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T12:24:49.974 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:49 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T12:24:49.974 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:49 smithi104 ceph-mon[25639]: mon.b calling monitor election 2024-09-21T12:24:49.974 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:49 smithi104 ceph-mon[25639]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:24:49.974 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:49 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T12:24:49.974 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:49 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T12:24:49.974 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:49 smithi104 ceph-mon[25639]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:24:49.975 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:49 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T12:24:49.975 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:49 smithi104 ceph-mon[25639]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-09-21T12:24:49.975 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:49 smithi104 ceph-mon[25639]: monmap e2: 2 mons at {a=[v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0],b=[v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-21T12:24:49.975 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:49 smithi104 ceph-mon[25639]: fsmap 2024-09-21T12:24:49.975 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:49 smithi104 ceph-mon[25639]: osdmap e4: 0 total, 0 up, 0 in 2024-09-21T12:24:49.975 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:49 smithi104 ceph-mon[25639]: mgrmap e13: y(active, since 38s) 2024-09-21T12:24:49.975 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:49 smithi104 ceph-mon[25639]: overall HEALTH_OK 2024-09-21T12:24:49.975 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:49 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:24:49.975 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:49 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:24:49.975 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:49 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:24:49.975 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:49 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-21T12:24:49.975 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:49 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:24:49.982 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-21T12:24:49.982 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":2,"fsid":"33168022-7814-11ef-baf6-efdc52797490","modified":"2024-09-21T12:24:44.657791Z","created":"2024-09-21T12:23:19.108132Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:3300","nonce":0},{"type":"v1","addr":"172.21.15.79:6789","nonce":0}]},"addr":"172.21.15.79:6789/0","public_addr":"172.21.15.79:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-09-21T12:24:49.982 INFO:teuthology.orchestra.run.smithi104.stderr:dumped monmap epoch 2 2024-09-21T12:24:50.051 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:49 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T12:24:50.051 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:49 smithi079 ceph-mon[23699]: mon.a calling monitor election 2024-09-21T12:24:50.051 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:49 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T12:24:50.051 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:49 smithi079 ceph-mon[23699]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:24:50.051 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:49 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T12:24:50.051 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:49 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T12:24:50.051 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:49 smithi079 ceph-mon[23699]: mon.b calling monitor election 2024-09-21T12:24:50.052 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:49 smithi079 ceph-mon[23699]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:24:50.052 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:49 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T12:24:50.052 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:49 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T12:24:50.052 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:49 smithi079 ceph-mon[23699]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:24:50.052 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:49 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T12:24:50.052 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:49 smithi079 ceph-mon[23699]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-09-21T12:24:50.052 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:49 smithi079 ceph-mon[23699]: monmap e2: 2 mons at {a=[v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0],b=[v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-21T12:24:50.052 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:49 smithi079 ceph-mon[23699]: fsmap 2024-09-21T12:24:50.052 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:49 smithi079 ceph-mon[23699]: osdmap e4: 0 total, 0 up, 0 in 2024-09-21T12:24:50.052 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:49 smithi079 ceph-mon[23699]: mgrmap e13: y(active, since 38s) 2024-09-21T12:24:50.052 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:49 smithi079 ceph-mon[23699]: overall HEALTH_OK 2024-09-21T12:24:50.052 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:49 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:24:50.052 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:49 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:24:50.053 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:49 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:24:50.053 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:49 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-21T12:24:50.053 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:49 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:24:50.916 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:50 smithi104 ceph-mon[25639]: Deploying daemon mon.c on smithi079 2024-09-21T12:24:50.916 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:50 smithi104 ceph-mon[25639]: from='client.? 172.21.15.104:0/4282049961' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-21T12:24:50.916 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:50 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T12:24:50.996 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:50 smithi079 ceph-mon[23699]: Deploying daemon mon.c on smithi079 2024-09-21T12:24:50.996 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:50 smithi079 ceph-mon[23699]: from='client.? 172.21.15.104:0/4282049961' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-21T12:24:50.996 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:50 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T12:24:51.546 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-21T12:24:51.547 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 33168022-7814-11ef-baf6-efdc52797490 -- ceph mon dump -f json 2024-09-21T12:24:51.740 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:24:51.862 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:51 smithi104 ceph-mon[25639]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:24:52.019 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:51 smithi079 ceph-mon[23699]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:24:52.647 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:52 smithi079 systemd[1]: Starting Ceph mon.c for 33168022-7814-11ef-baf6-efdc52797490... 2024-09-21T12:24:52.942 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:52 smithi079 podman[30005]: 2024-09-21 12:24:52.548180498 +0000 UTC m=+0.015719483 image pull 0adab276c97e6f3143ffbca5eeff37fcd7dc3b5c0f339a9f747381d9993602de quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6 2024-09-21T12:24:52.942 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:52 smithi079 podman[30005]: 2024-09-21 12:24:52.65366837 +0000 UTC m=+0.121207401 container create 4b837a3d8ec3f5a95e11dac3edadd2bfc94e662bcb04ab8dc8360ea3677dc85c (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-33168022-7814-11ef-baf6-efdc52797490-mon-c, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, ceph=True, io.buildah.version=1.37.2, GIT_CLEAN=True, RELEASE=reef-6a1bdf8, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T12:24:53.443 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 podman[30005]: 2024-09-21 12:24:53.01492069 +0000 UTC m=+0.482459679 container init 4b837a3d8ec3f5a95e11dac3edadd2bfc94e662bcb04ab8dc8360ea3677dc85c (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-33168022-7814-11ef-baf6-efdc52797490-mon-c, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, RELEASE=reef-6a1bdf8, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T12:24:53.443 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 podman[30005]: 2024-09-21 12:24:53.018191417 +0000 UTC m=+0.485730402 container start 4b837a3d8ec3f5a95e11dac3edadd2bfc94e662bcb04ab8dc8360ea3677dc85c (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-33168022-7814-11ef-baf6-efdc52797490-mon-c, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, RELEASE=reef-6a1bdf8, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-09-21T12:24:53.443 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: set uid:gid to 167:167 (ceph:ceph) 2024-09-21T12:24:53.443 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: ceph version 18.2.4-871-g6a1bdf85 (6a1bdf854bb5803efd58957907442aa01f41a8fc) reef (stable), process ceph-mon, pid 2 2024-09-21T12:24:53.443 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: pidfile_write: ignore empty --pid-file 2024-09-21T12:24:53.443 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: load: jerasure load: lrc 2024-09-21T12:24:53.443 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: RocksDB version: 7.9.2 2024-09-21T12:24:53.443 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Git sha 0 2024-09-21T12:24:53.443 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Compile date 2024-09-20 20:34:00 2024-09-21T12:24:53.443 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: DB SUMMARY 2024-09-21T12:24:53.444 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: DB Session ID: C5R5QTV33GJN62OLIWG0 2024-09-21T12:24:53.444 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: CURRENT file: CURRENT 2024-09-21T12:24:53.444 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: IDENTITY file: IDENTITY 2024-09-21T12:24:53.444 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-09-21T12:24:53.444 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-09-21T12:24:53.444 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 636 ; 2024-09-21T12:24:53.444 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.error_if_exists: 0 2024-09-21T12:24:53.444 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.create_if_missing: 0 2024-09-21T12:24:53.444 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.paranoid_checks: 1 2024-09-21T12:24:53.444 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.flush_verify_memtable_count: 1 2024-09-21T12:24:53.444 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-21T12:24:53.444 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-09-21T12:24:53.444 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.env: 0x5609fa9a7bc0 2024-09-21T12:24:53.444 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.fs: PosixFileSystem 2024-09-21T12:24:53.444 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.info_log: 0x5609fc2eabe0 2024-09-21T12:24:53.445 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.max_file_opening_threads: 16 2024-09-21T12:24:53.445 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.statistics: (nil) 2024-09-21T12:24:53.446 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.use_fsync: 0 2024-09-21T12:24:53.446 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.max_log_file_size: 0 2024-09-21T12:24:53.446 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-21T12:24:53.446 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-21T12:24:53.446 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.keep_log_file_num: 1000 2024-09-21T12:24:53.446 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.recycle_log_file_num: 0 2024-09-21T12:24:53.446 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.allow_fallocate: 1 2024-09-21T12:24:53.446 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.allow_mmap_reads: 0 2024-09-21T12:24:53.446 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.allow_mmap_writes: 0 2024-09-21T12:24:53.446 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.use_direct_reads: 0 2024-09-21T12:24:53.446 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-21T12:24:53.447 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.create_missing_column_families: 0 2024-09-21T12:24:53.447 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.db_log_dir: 2024-09-21T12:24:53.447 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.wal_dir: 2024-09-21T12:24:53.447 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-21T12:24:53.447 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-21T12:24:53.447 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-21T12:24:53.447 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-21T12:24:53.447 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-21T12:24:53.447 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-21T12:24:53.447 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.advise_random_on_open: 1 2024-09-21T12:24:53.447 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.db_write_buffer_size: 0 2024-09-21T12:24:53.447 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.write_buffer_manager: 0x5609fc2f8320 2024-09-21T12:24:53.447 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-21T12:24:53.447 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-21T12:24:53.447 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-21T12:24:53.448 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.rate_limiter: (nil) 2024-09-21T12:24:53.448 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-21T12:24:53.448 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.wal_recovery_mode: 2 2024-09-21T12:24:53.448 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.enable_thread_tracking: 0 2024-09-21T12:24:53.448 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.enable_pipelined_write: 0 2024-09-21T12:24:53.448 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.unordered_write: 0 2024-09-21T12:24:53.448 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-21T12:24:53.448 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-21T12:24:53.448 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-21T12:24:53.448 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-21T12:24:53.448 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.row_cache: None 2024-09-21T12:24:53.448 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.wal_filter: None 2024-09-21T12:24:53.448 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-21T12:24:53.448 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.allow_ingest_behind: 0 2024-09-21T12:24:53.449 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.two_write_queues: 0 2024-09-21T12:24:53.449 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.manual_wal_flush: 0 2024-09-21T12:24:53.449 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.wal_compression: 0 2024-09-21T12:24:53.449 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.atomic_flush: 0 2024-09-21T12:24:53.449 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-21T12:24:53.449 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-21T12:24:53.449 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-21T12:24:53.449 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.log_readahead_size: 0 2024-09-21T12:24:53.449 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-21T12:24:53.449 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.best_efforts_recovery: 0 2024-09-21T12:24:53.449 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-21T12:24:53.449 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-21T12:24:53.449 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.allow_data_in_errors: 0 2024-09-21T12:24:53.449 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.db_host_id: __hostname__ 2024-09-21T12:24:53.449 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.enforce_single_del_contracts: true 2024-09-21T12:24:53.449 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.max_background_jobs: 2 2024-09-21T12:24:53.450 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.max_background_compactions: -1 2024-09-21T12:24:53.450 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.max_subcompactions: 1 2024-09-21T12:24:53.450 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-21T12:24:53.450 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-21T12:24:53.450 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-21T12:24:53.450 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.max_total_wal_size: 0 2024-09-21T12:24:53.450 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-21T12:24:53.450 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-21T12:24:53.450 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-21T12:24:53.450 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-21T12:24:53.450 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.max_open_files: -1 2024-09-21T12:24:53.450 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.bytes_per_sync: 0 2024-09-21T12:24:53.450 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-21T12:24:53.450 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-21T12:24:53.451 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.compaction_readahead_size: 0 2024-09-21T12:24:53.451 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.max_background_flushes: -1 2024-09-21T12:24:53.451 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Compression algorithms supported: 2024-09-21T12:24:53.451 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: kZSTD supported: 0 2024-09-21T12:24:53.451 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: kXpressCompression supported: 0 2024-09-21T12:24:53.451 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: kBZip2Compression supported: 0 2024-09-21T12:24:53.451 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-21T12:24:53.451 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: kLZ4Compression supported: 1 2024-09-21T12:24:53.451 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: kZlibCompression supported: 1 2024-09-21T12:24:53.451 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: kLZ4HCCompression supported: 1 2024-09-21T12:24:53.451 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: kSnappyCompression supported: 1 2024-09-21T12:24:53.451 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-21T12:24:53.451 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: DMutex implementation: pthread_mutex_t 2024-09-21T12:24:53.451 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-09-21T12:24:53.452 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-09-21T12:24:53.452 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-21T12:24:53.452 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.merge_operator: 2024-09-21T12:24:53.452 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.compaction_filter: None 2024-09-21T12:24:53.452 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.compaction_filter_factory: None 2024-09-21T12:24:53.452 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.sst_partitioner_factory: None 2024-09-21T12:24:53.452 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-21T12:24:53.452 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-21T12:24:53.452 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5609fc2ea7e0) 2024-09-21T12:24:53.452 INFO:journalctl@ceph.mon.c.smithi079.stdout: cache_index_and_filter_blocks: 1 2024-09-21T12:24:53.452 INFO:journalctl@ceph.mon.c.smithi079.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-21T12:24:53.452 INFO:journalctl@ceph.mon.c.smithi079.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-21T12:24:53.452 INFO:journalctl@ceph.mon.c.smithi079.stdout: pin_top_level_index_and_filter: 1 2024-09-21T12:24:53.452 INFO:journalctl@ceph.mon.c.smithi079.stdout: index_type: 0 2024-09-21T12:24:53.452 INFO:journalctl@ceph.mon.c.smithi079.stdout: data_block_index_type: 0 2024-09-21T12:24:53.452 INFO:journalctl@ceph.mon.c.smithi079.stdout: index_shortening: 1 2024-09-21T12:24:53.453 INFO:journalctl@ceph.mon.c.smithi079.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-21T12:24:53.453 INFO:journalctl@ceph.mon.c.smithi079.stdout: checksum: 4 2024-09-21T12:24:53.453 INFO:journalctl@ceph.mon.c.smithi079.stdout: no_block_cache: 0 2024-09-21T12:24:53.453 INFO:journalctl@ceph.mon.c.smithi079.stdout: block_cache: 0x5609fc2df1f0 2024-09-21T12:24:53.453 INFO:journalctl@ceph.mon.c.smithi079.stdout: block_cache_name: BinnedLRUCache 2024-09-21T12:24:53.453 INFO:journalctl@ceph.mon.c.smithi079.stdout: block_cache_options: 2024-09-21T12:24:53.453 INFO:journalctl@ceph.mon.c.smithi079.stdout: capacity : 536870912 2024-09-21T12:24:53.453 INFO:journalctl@ceph.mon.c.smithi079.stdout: num_shard_bits : 4 2024-09-21T12:24:53.453 INFO:journalctl@ceph.mon.c.smithi079.stdout: strict_capacity_limit : 0 2024-09-21T12:24:53.453 INFO:journalctl@ceph.mon.c.smithi079.stdout: high_pri_pool_ratio: 0.000 2024-09-21T12:24:53.453 INFO:journalctl@ceph.mon.c.smithi079.stdout: block_cache_compressed: (nil) 2024-09-21T12:24:53.453 INFO:journalctl@ceph.mon.c.smithi079.stdout: persistent_cache: (nil) 2024-09-21T12:24:53.453 INFO:journalctl@ceph.mon.c.smithi079.stdout: block_size: 4096 2024-09-21T12:24:53.453 INFO:journalctl@ceph.mon.c.smithi079.stdout: block_size_deviation: 10 2024-09-21T12:24:53.453 INFO:journalctl@ceph.mon.c.smithi079.stdout: block_restart_interval: 16 2024-09-21T12:24:53.454 INFO:journalctl@ceph.mon.c.smithi079.stdout: index_block_restart_interval: 1 2024-09-21T12:24:53.454 INFO:journalctl@ceph.mon.c.smithi079.stdout: metadata_block_size: 4096 2024-09-21T12:24:53.454 INFO:journalctl@ceph.mon.c.smithi079.stdout: partition_filters: 0 2024-09-21T12:24:53.454 INFO:journalctl@ceph.mon.c.smithi079.stdout: use_delta_encoding: 1 2024-09-21T12:24:53.454 INFO:journalctl@ceph.mon.c.smithi079.stdout: filter_policy: bloomfilter 2024-09-21T12:24:53.454 INFO:journalctl@ceph.mon.c.smithi079.stdout: whole_key_filtering: 1 2024-09-21T12:24:53.454 INFO:journalctl@ceph.mon.c.smithi079.stdout: verify_compression: 0 2024-09-21T12:24:53.454 INFO:journalctl@ceph.mon.c.smithi079.stdout: read_amp_bytes_per_bit: 0 2024-09-21T12:24:53.454 INFO:journalctl@ceph.mon.c.smithi079.stdout: format_version: 5 2024-09-21T12:24:53.454 INFO:journalctl@ceph.mon.c.smithi079.stdout: enable_index_compression: 1 2024-09-21T12:24:53.454 INFO:journalctl@ceph.mon.c.smithi079.stdout: block_align: 0 2024-09-21T12:24:53.454 INFO:journalctl@ceph.mon.c.smithi079.stdout: max_auto_readahead_size: 262144 2024-09-21T12:24:53.454 INFO:journalctl@ceph.mon.c.smithi079.stdout: prepopulate_block_cache: 0 2024-09-21T12:24:53.454 INFO:journalctl@ceph.mon.c.smithi079.stdout: initial_auto_readahead_size: 8192 2024-09-21T12:24:53.454 INFO:journalctl@ceph.mon.c.smithi079.stdout: num_file_reads_for_auto_readahead: 2 2024-09-21T12:24:53.454 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.write_buffer_size: 33554432 2024-09-21T12:24:53.455 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.max_write_buffer_number: 2 2024-09-21T12:24:53.455 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.compression: NoCompression 2024-09-21T12:24:53.455 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.bottommost_compression: Disabled 2024-09-21T12:24:53.455 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.prefix_extractor: nullptr 2024-09-21T12:24:53.455 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-21T12:24:53.455 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.num_levels: 7 2024-09-21T12:24:53.455 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-21T12:24:53.455 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-21T12:24:53.455 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-21T12:24:53.455 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-21T12:24:53.455 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-21T12:24:53.455 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-21T12:24:53.455 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-21T12:24:53.455 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-21T12:24:53.456 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-21T12:24:53.456 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-21T12:24:53.456 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-09-21T12:24:53.456 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-09-21T12:24:53.456 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-21T12:24:53.456 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.compression_opts.level: 32767 2024-09-21T12:24:53.456 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.compression_opts.strategy: 0 2024-09-21T12:24:53.456 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-21T12:24:53.456 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-21T12:24:53.456 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-09-21T12:24:53.456 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-21T12:24:53.456 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.compression_opts.enabled: false 2024-09-21T12:24:53.456 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-09-21T12:24:53.456 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-21T12:24:53.456 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-21T12:24:53.456 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-21T12:24:53.457 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.target_file_size_base: 67108864 2024-09-21T12:24:53.457 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-21T12:24:53.457 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-21T12:24:53.457 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-21T12:24:53.457 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-21T12:24:53.457 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-21T12:24:53.457 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-21T12:24:53.457 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-21T12:24:53.457 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-21T12:24:53.457 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-21T12:24:53.457 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-21T12:24:53.457 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-21T12:24:53.457 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-21T12:24:53.457 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-21T12:24:53.457 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-09-21T12:24:53.457 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.arena_block_size: 1048576 2024-09-21T12:24:53.458 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-21T12:24:53.458 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-21T12:24:53.458 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.disable_auto_compactions: 0 2024-09-21T12:24:53.458 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-21T12:24:53.458 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-21T12:24:53.458 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-21T12:24:53.458 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-21T12:24:53.458 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-21T12:24:53.458 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-21T12:24:53.458 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-21T12:24:53.458 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-21T12:24:53.458 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-21T12:24:53.458 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-21T12:24:53.458 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-21T12:24:53.459 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.inplace_update_support: 0 2024-09-21T12:24:53.459 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-21T12:24:53.459 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-21T12:24:53.459 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-21T12:24:53.459 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-21T12:24:53.459 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.bloom_locality: 0 2024-09-21T12:24:53.459 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.max_successive_merges: 0 2024-09-21T12:24:53.459 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-21T12:24:53.459 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.paranoid_file_checks: 0 2024-09-21T12:24:53.459 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.force_consistency_checks: 1 2024-09-21T12:24:53.459 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.report_bg_io_stats: 0 2024-09-21T12:24:53.459 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.ttl: 2592000 2024-09-21T12:24:53.459 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-21T12:24:53.459 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-09-21T12:24:53.459 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-09-21T12:24:53.460 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.enable_blob_files: false 2024-09-21T12:24:53.460 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.min_blob_size: 0 2024-09-21T12:24:53.460 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.blob_file_size: 268435456 2024-09-21T12:24:53.460 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-21T12:24:53.460 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-21T12:24:53.460 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-21T12:24:53.460 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-09-21T12:24:53.460 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-09-21T12:24:53.460 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.blob_file_starting_level: 0 2024-09-21T12:24:53.460 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-09-21T12:24:53.460 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-09-21T12:24:53.460 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-09-21T12:24:53.460 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: cf3c2452-67a8-47a7-bc09-5b7e3a9a0947 2024-09-21T12:24:53.460 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726921493044418, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-09-21T12:24:53.460 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-09-21T12:24:53.461 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726921493044785, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1773, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1726921493, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "cf3c2452-67a8-47a7-bc09-5b7e3a9a0947", "db_session_id": "C5R5QTV33GJN62OLIWG0", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-09-21T12:24:53.461 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726921493044876, "job": 1, "event": "recovery_finished"} 2024-09-21T12:24:53.461 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-09-21T12:24:53.461 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-21T12:24:53.461 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5609fc30ae00 2024-09-21T12:24:53.461 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: DB pointer 0x5609fc3fe000 2024-09-21T12:24:53.461 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-09-21T12:24:53.461 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-09-21T12:24:53.461 INFO:journalctl@ceph.mon.c.smithi079.stdout: ** DB Stats ** 2024-09-21T12:24:53.461 INFO:journalctl@ceph.mon.c.smithi079.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-21T12:24:53.461 INFO:journalctl@ceph.mon.c.smithi079.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-21T12:24:53.461 INFO:journalctl@ceph.mon.c.smithi079.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-21T12:24:53.461 INFO:journalctl@ceph.mon.c.smithi079.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-21T12:24:53.461 INFO:journalctl@ceph.mon.c.smithi079.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-21T12:24:53.461 INFO:journalctl@ceph.mon.c.smithi079.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-21T12:24:53.461 INFO:journalctl@ceph.mon.c.smithi079.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-21T12:24:53.462 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2024-09-21T12:24:53.462 INFO:journalctl@ceph.mon.c.smithi079.stdout: ** Compaction Stats [default] ** 2024-09-21T12:24:53.462 INFO:journalctl@ceph.mon.c.smithi079.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-21T12:24:53.462 INFO:journalctl@ceph.mon.c.smithi079.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-09-21T12:24:53.462 INFO:journalctl@ceph.mon.c.smithi079.stdout: L0 1/0 1.73 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.8 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-21T12:24:53.462 INFO:journalctl@ceph.mon.c.smithi079.stdout: Sum 1/0 1.73 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.8 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-21T12:24:53.462 INFO:journalctl@ceph.mon.c.smithi079.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.8 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-21T12:24:53.462 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2024-09-21T12:24:53.462 INFO:journalctl@ceph.mon.c.smithi079.stdout: ** Compaction Stats [default] ** 2024-09-21T12:24:53.462 INFO:journalctl@ceph.mon.c.smithi079.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-21T12:24:53.462 INFO:journalctl@ceph.mon.c.smithi079.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-21T12:24:53.462 INFO:journalctl@ceph.mon.c.smithi079.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.8 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-21T12:24:53.462 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2024-09-21T12:24:53.462 INFO:journalctl@ceph.mon.c.smithi079.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-09-21T12:24:53.463 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2024-09-21T12:24:53.463 INFO:journalctl@ceph.mon.c.smithi079.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-21T12:24:53.463 INFO:journalctl@ceph.mon.c.smithi079.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-21T12:24:53.463 INFO:journalctl@ceph.mon.c.smithi079.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-21T12:24:53.463 INFO:journalctl@ceph.mon.c.smithi079.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-21T12:24:53.463 INFO:journalctl@ceph.mon.c.smithi079.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-21T12:24:53.463 INFO:journalctl@ceph.mon.c.smithi079.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-21T12:24:53.463 INFO:journalctl@ceph.mon.c.smithi079.stdout: Cumulative compaction: 0.00 GB write, 0.69 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-21T12:24:53.464 INFO:journalctl@ceph.mon.c.smithi079.stdout: Interval compaction: 0.00 GB write, 0.69 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-21T12:24:53.464 INFO:journalctl@ceph.mon.c.smithi079.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-21T12:24:53.464 INFO:journalctl@ceph.mon.c.smithi079.stdout: Block cache BinnedLRUCache@0x5609fc2df1f0#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-09-21T12:24:53.464 INFO:journalctl@ceph.mon.c.smithi079.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-09-21T12:24:53.464 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2024-09-21T12:24:53.464 INFO:journalctl@ceph.mon.c.smithi079.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-21T12:24:53.464 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-09-21T12:24:53.464 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: using public_addrv [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] 2024-09-21T12:24:53.464 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: starting mon.c rank -1 at public addrs [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] at bind addrs [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] mon_data /var/lib/ceph/mon/ceph-c fsid 33168022-7814-11ef-baf6-efdc52797490 2024-09-21T12:24:53.464 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: mon.c@-1(???) e0 preinit fsid 33168022-7814-11ef-baf6-efdc52797490 2024-09-21T12:24:53.464 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: mon.c@-1(synchronizing).mds e1 new map 2024-09-21T12:24:53.465 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: mon.c@-1(synchronizing).mds e1 print_map 2024-09-21T12:24:53.465 INFO:journalctl@ceph.mon.c.smithi079.stdout: e1 2024-09-21T12:24:53.465 INFO:journalctl@ceph.mon.c.smithi079.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-21T12:24:53.465 INFO:journalctl@ceph.mon.c.smithi079.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-09-21T12:24:53.465 INFO:journalctl@ceph.mon.c.smithi079.stdout: legacy client fscid: -1 2024-09-21T12:24:53.465 INFO:journalctl@ceph.mon.c.smithi079.stdout: 2024-09-21T12:24:53.465 INFO:journalctl@ceph.mon.c.smithi079.stdout: No filesystems configured 2024-09-21T12:24:53.465 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-09-21T12:24:53.465 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-09-21T12:24:53.465 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-09-21T12:24:53.465 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-09-21T12:24:53.465 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-09-21T12:24:53.465 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-09-21T12:24:53.465 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-09-21T12:24:53.465 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-21T12:24:53.466 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-21T12:24:53.466 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-21T12:24:53.466 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: Deploying daemon agent.smithi104 on smithi104 2024-09-21T12:24:53.466 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T12:24:53.466 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:24:53.466 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:24:53.466 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:24:53.466 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:24:53.466 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:24:53.466 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:24:53.466 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:24:53.466 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:24:53.466 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:24:53.466 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:24:53.467 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:24:53.467 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:24:53.467 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:24:53.467 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:24:53.467 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:24:53.467 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:24:53.467 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: from='client.? 172.21.15.79:0/3618158966' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-09-21T12:24:53.467 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:24:53.467 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: from='client.? 172.21.15.79:0/3618158966' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-09-21T12:24:53.467 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: osdmap e4: 0 total, 0 up, 0 in 2024-09-21T12:24:53.467 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:24:53.467 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:24:53.467 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:24:53.467 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:24:53.467 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:24:53.468 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:24:53.468 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:24:53.468 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:24:53.468 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-21T12:24:53.468 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:24:53.468 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi079:172.21.15.79=a;smithi079:[v2:172.21.15.79:3301,v1:172.21.15.79:6790]=c;smithi104:172.21.15.104=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:24:53.468 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: Saving service mon spec with placement smithi079:172.21.15.79=a;smithi079:[v2:172.21.15.79:3301,v1:172.21.15.79:6790]=c;smithi104:172.21.15.104=b;count:3 2024-09-21T12:24:53.468 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: Deploying daemon mon.b on smithi104 2024-09-21T12:24:53.468 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:24:53.468 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:24:53.468 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:24:53.468 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:24:53.468 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:24:53.468 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: from='client.? 172.21.15.104:0/4019363617' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-21T12:24:53.469 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T12:24:53.469 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: mon.a calling monitor election 2024-09-21T12:24:53.469 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T12:24:53.469 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:24:53.469 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T12:24:53.469 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T12:24:53.469 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: mon.b calling monitor election 2024-09-21T12:24:53.469 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:24:53.469 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T12:24:53.469 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T12:24:53.469 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:24:53.469 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T12:24:53.469 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-09-21T12:24:53.469 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: monmap e2: 2 mons at {a=[v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0],b=[v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-21T12:24:53.469 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: fsmap 2024-09-21T12:24:53.470 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: osdmap e4: 0 total, 0 up, 0 in 2024-09-21T12:24:53.470 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: mgrmap e13: y(active, since 38s) 2024-09-21T12:24:53.470 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: overall HEALTH_OK 2024-09-21T12:24:53.470 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:24:53.470 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:24:53.470 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:24:53.470 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-21T12:24:53.470 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:24:53.470 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: Deploying daemon mon.c on smithi079 2024-09-21T12:24:53.470 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: from='client.? 172.21.15.104:0/4282049961' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-21T12:24:53.470 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T12:24:53.470 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:24:53.470 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 ceph-mon[30023]: mon.c@-1(synchronizing).paxosservice(auth 1..5) refresh upgraded, format 0 -> 3 2024-09-21T12:24:53.470 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 bash[30005]: 4b837a3d8ec3f5a95e11dac3edadd2bfc94e662bcb04ab8dc8360ea3677dc85c 2024-09-21T12:24:53.471 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:24:53 smithi079 systemd[1]: Started Ceph mon.c for 33168022-7814-11ef-baf6-efdc52797490. 2024-09-21T12:24:58.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:58 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T12:24:58.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:58 smithi104 ceph-mon[25639]: mon.a calling monitor election 2024-09-21T12:24:58.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:58 smithi104 ceph-mon[25639]: mon.b calling monitor election 2024-09-21T12:24:58.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:58 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T12:24:58.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:58 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T12:24:58.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:58 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T12:24:58.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:58 smithi104 ceph-mon[25639]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:24:58.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:58 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T12:24:58.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:58 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T12:24:58.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:58 smithi104 ceph-mon[25639]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:24:58.428 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:58 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T12:24:58.428 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:58 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T12:24:58.428 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:58 smithi104 ceph-mon[25639]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-09-21T12:24:58.428 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:58 smithi104 ceph-mon[25639]: monmap e3: 3 mons at {a=[v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0],b=[v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0],c=[v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-21T12:24:58.428 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:58 smithi104 ceph-mon[25639]: fsmap 2024-09-21T12:24:58.428 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:58 smithi104 ceph-mon[25639]: osdmap e4: 0 total, 0 up, 0 in 2024-09-21T12:24:58.428 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:58 smithi104 ceph-mon[25639]: mgrmap e13: y(active, since 47s) 2024-09-21T12:24:58.428 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:58 smithi104 ceph-mon[25639]: Health check failed: 1/3 mons down, quorum a,b (MON_DOWN) 2024-09-21T12:24:58.428 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:58 smithi104 ceph-mon[25639]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,b 2024-09-21T12:24:58.428 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:58 smithi104 ceph-mon[25639]: [WRN] MON_DOWN: 1/3 mons down, quorum a,b 2024-09-21T12:24:58.428 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:58 smithi104 ceph-mon[25639]: mon.c (rank 2) addr [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] is down (out of quorum) 2024-09-21T12:24:58.428 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:58 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:24:58.429 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:58 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:24:58.429 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:58 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:24:58.429 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:58 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:24:58.429 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:58 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:24:58.429 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:58 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:24:58.429 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:58 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:24:58.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:58 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T12:24:58.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:58 smithi079 ceph-mon[23699]: mon.a calling monitor election 2024-09-21T12:24:58.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:58 smithi079 ceph-mon[23699]: mon.b calling monitor election 2024-09-21T12:24:58.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:58 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T12:24:58.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:58 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T12:24:58.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:58 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T12:24:58.443 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:58 smithi079 ceph-mon[23699]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:24:58.443 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:58 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T12:24:58.443 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:58 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T12:24:58.443 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:58 smithi079 ceph-mon[23699]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:24:58.443 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:58 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T12:24:58.443 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:58 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T12:24:58.443 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:58 smithi079 ceph-mon[23699]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-09-21T12:24:58.443 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:58 smithi079 ceph-mon[23699]: monmap e3: 3 mons at {a=[v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0],b=[v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0],c=[v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-21T12:24:58.443 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:58 smithi079 ceph-mon[23699]: fsmap 2024-09-21T12:24:58.443 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:58 smithi079 ceph-mon[23699]: osdmap e4: 0 total, 0 up, 0 in 2024-09-21T12:24:58.443 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:58 smithi079 ceph-mon[23699]: mgrmap e13: y(active, since 47s) 2024-09-21T12:24:58.443 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:58 smithi079 ceph-mon[23699]: Health check failed: 1/3 mons down, quorum a,b (MON_DOWN) 2024-09-21T12:24:58.443 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:58 smithi079 ceph-mon[23699]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,b 2024-09-21T12:24:58.443 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:58 smithi079 ceph-mon[23699]: [WRN] MON_DOWN: 1/3 mons down, quorum a,b 2024-09-21T12:24:58.443 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:58 smithi079 ceph-mon[23699]: mon.c (rank 2) addr [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] is down (out of quorum) 2024-09-21T12:24:58.443 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:58 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:24:58.444 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:58 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:24:58.444 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:58 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:24:58.444 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:58 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:24:58.444 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:58 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:24:58.444 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:58 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:24:58.444 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:58 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:24:59.309 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:59 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:24:59.309 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:59 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:24:59.310 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:59 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:24:59.310 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:59 smithi079 ceph-mon[23699]: Updating smithi079:/etc/ceph/ceph.conf 2024-09-21T12:24:59.310 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:59 smithi079 ceph-mon[23699]: Updating smithi104:/etc/ceph/ceph.conf 2024-09-21T12:24:59.310 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:59 smithi079 ceph-mon[23699]: Updating smithi079:/var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T12:24:59.310 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:59 smithi079 ceph-mon[23699]: Updating smithi104:/var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T12:24:59.310 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:59 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:24:59.310 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:59 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:24:59.310 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:59 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:24:59.310 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:59 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:24:59.310 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:59 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:24:59.310 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:59 smithi079 ceph-mon[23699]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-21T12:24:59.310 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:59 smithi079 ceph-mon[23699]: Reconfiguring mon.a (unknown last config time)... 2024-09-21T12:24:59.310 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:59 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-21T12:24:59.311 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:59 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-21T12:24:59.311 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:59 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:24:59.311 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:59 smithi079 ceph-mon[23699]: Reconfiguring daemon mon.a on smithi079 2024-09-21T12:24:59.311 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:59 smithi079 ceph-mon[23699]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:24:59.311 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:24:59 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T12:24:59.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:59 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:24:59.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:59 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:24:59.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:59 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:24:59.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:59 smithi104 ceph-mon[25639]: Updating smithi079:/etc/ceph/ceph.conf 2024-09-21T12:24:59.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:59 smithi104 ceph-mon[25639]: Updating smithi104:/etc/ceph/ceph.conf 2024-09-21T12:24:59.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:59 smithi104 ceph-mon[25639]: Updating smithi079:/var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T12:24:59.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:59 smithi104 ceph-mon[25639]: Updating smithi104:/var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T12:24:59.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:59 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:24:59.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:59 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:24:59.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:59 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:24:59.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:59 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:24:59.428 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:59 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:24:59.428 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:59 smithi104 ceph-mon[25639]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-21T12:24:59.428 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:59 smithi104 ceph-mon[25639]: Reconfiguring mon.a (unknown last config time)... 2024-09-21T12:24:59.428 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:59 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-21T12:24:59.428 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:59 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-21T12:24:59.428 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:59 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:24:59.428 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:59 smithi104 ceph-mon[25639]: Reconfiguring daemon mon.a on smithi079 2024-09-21T12:24:59.428 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:59 smithi104 ceph-mon[25639]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:24:59.428 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:24:59 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T12:24:59.470 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-21T12:24:59.470 INFO:teuthology.orchestra.run.smithi104.stdout:{"epoch":3,"fsid":"33168022-7814-11ef-baf6-efdc52797490","modified":"2024-09-21T12:24:53.068769Z","created":"2024-09-21T12:23:19.108132Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:3300","nonce":0},{"type":"v1","addr":"172.21.15.79:6789","nonce":0}]},"addr":"172.21.15.79:6789/0","public_addr":"172.21.15.79:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:3300","nonce":0},{"type":"v1","addr":"172.21.15.104:6789","nonce":0}]},"addr":"172.21.15.104:6789/0","public_addr":"172.21.15.104:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:3301","nonce":0},{"type":"v1","addr":"172.21.15.79:6790","nonce":0}]},"addr":"172.21.15.79:6790/0","public_addr":"172.21.15.79:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-09-21T12:24:59.470 INFO:teuthology.orchestra.run.smithi104.stderr:dumped monmap epoch 3 2024-09-21T12:25:00.193 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:00 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T12:25:00.193 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:00 smithi079 ceph-mon[30023]: mon.a calling monitor election 2024-09-21T12:25:00.193 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:00 smithi079 ceph-mon[30023]: mon.b calling monitor election 2024-09-21T12:25:00.193 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:00 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T12:25:00.193 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:00 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T12:25:00.194 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:00 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T12:25:00.194 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:00 smithi079 ceph-mon[30023]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:25:00.194 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:00 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T12:25:00.194 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:00 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T12:25:00.194 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:00 smithi079 ceph-mon[30023]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:25:00.194 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:00 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T12:25:00.194 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:00 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T12:25:00.194 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:00 smithi079 ceph-mon[30023]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-09-21T12:25:00.194 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:00 smithi079 ceph-mon[30023]: monmap e3: 3 mons at {a=[v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0],b=[v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0],c=[v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-21T12:25:00.194 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:00 smithi079 ceph-mon[30023]: fsmap 2024-09-21T12:25:00.194 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:00 smithi079 ceph-mon[30023]: osdmap e4: 0 total, 0 up, 0 in 2024-09-21T12:25:00.194 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:00 smithi079 ceph-mon[30023]: mgrmap e13: y(active, since 47s) 2024-09-21T12:25:00.194 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:00 smithi079 ceph-mon[30023]: Health check failed: 1/3 mons down, quorum a,b (MON_DOWN) 2024-09-21T12:25:00.194 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:00 smithi079 ceph-mon[30023]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,b 2024-09-21T12:25:00.195 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:00 smithi079 ceph-mon[30023]: [WRN] MON_DOWN: 1/3 mons down, quorum a,b 2024-09-21T12:25:00.195 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:00 smithi079 ceph-mon[30023]: mon.c (rank 2) addr [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] is down (out of quorum) 2024-09-21T12:25:00.195 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:00 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:00.195 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:00 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:00.195 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:00 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:00.195 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:00 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:00.195 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:00 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:00.195 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:00 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:00.195 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:00 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:00.195 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:00 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:25:00.195 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:00 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:25:00.195 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:00 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:25:00.195 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:00 smithi079 ceph-mon[30023]: Updating smithi079:/etc/ceph/ceph.conf 2024-09-21T12:25:00.195 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:00 smithi079 ceph-mon[30023]: Updating smithi104:/etc/ceph/ceph.conf 2024-09-21T12:25:00.195 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:00 smithi079 ceph-mon[30023]: Updating smithi079:/var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T12:25:00.195 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:00 smithi079 ceph-mon[30023]: Updating smithi104:/var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T12:25:00.196 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:00 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:00.196 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:00 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:00.196 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:00 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:00.196 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:00 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:00.196 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:00 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:00.196 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:00 smithi079 ceph-mon[30023]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-21T12:25:00.196 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:00 smithi079 ceph-mon[30023]: Reconfiguring mon.a (unknown last config time)... 2024-09-21T12:25:00.196 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:00 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-21T12:25:00.196 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:00 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-21T12:25:00.196 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:00 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:25:00.196 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:00 smithi079 ceph-mon[30023]: Reconfiguring daemon mon.a on smithi079 2024-09-21T12:25:00.196 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:00 smithi079 ceph-mon[30023]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:25:00.196 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:00 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T12:25:00.564 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-09-21T12:25:00.564 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 33168022-7814-11ef-baf6-efdc52797490 -- ceph config generate-minimal-conf 2024-09-21T12:25:00.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:00 smithi104 ceph-mon[25639]: mon.c calling monitor election 2024-09-21T12:25:00.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:00 smithi104 ceph-mon[25639]: mon.c calling monitor election 2024-09-21T12:25:00.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:00 smithi104 ceph-mon[25639]: mon.a calling monitor election 2024-09-21T12:25:00.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:00 smithi104 ceph-mon[25639]: mon.b calling monitor election 2024-09-21T12:25:00.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:00 smithi104 ceph-mon[25639]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-21T12:25:00.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:00 smithi104 ceph-mon[25639]: monmap e3: 3 mons at {a=[v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0],b=[v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0],c=[v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-21T12:25:00.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:00 smithi104 ceph-mon[25639]: fsmap 2024-09-21T12:25:00.928 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:00 smithi104 ceph-mon[25639]: osdmap e4: 0 total, 0 up, 0 in 2024-09-21T12:25:00.928 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:00 smithi104 ceph-mon[25639]: mgrmap e13: y(active, since 49s) 2024-09-21T12:25:00.928 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:00 smithi104 ceph-mon[25639]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,b) 2024-09-21T12:25:00.928 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:00 smithi104 ceph-mon[25639]: Cluster is now healthy 2024-09-21T12:25:00.928 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:00 smithi104 ceph-mon[25639]: overall HEALTH_OK 2024-09-21T12:25:00.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:00 smithi079 ceph-mon[23699]: mon.c calling monitor election 2024-09-21T12:25:00.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:00 smithi079 ceph-mon[23699]: mon.c calling monitor election 2024-09-21T12:25:00.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:00 smithi079 ceph-mon[23699]: mon.a calling monitor election 2024-09-21T12:25:00.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:00 smithi079 ceph-mon[23699]: mon.b calling monitor election 2024-09-21T12:25:00.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:00 smithi079 ceph-mon[23699]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-21T12:25:00.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:00 smithi079 ceph-mon[23699]: monmap e3: 3 mons at {a=[v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0],b=[v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0],c=[v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-21T12:25:00.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:00 smithi079 ceph-mon[23699]: fsmap 2024-09-21T12:25:00.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:00 smithi079 ceph-mon[23699]: osdmap e4: 0 total, 0 up, 0 in 2024-09-21T12:25:00.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:00 smithi079 ceph-mon[23699]: mgrmap e13: y(active, since 49s) 2024-09-21T12:25:00.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:00 smithi079 ceph-mon[23699]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,b) 2024-09-21T12:25:00.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:00 smithi079 ceph-mon[23699]: Cluster is now healthy 2024-09-21T12:25:00.943 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:00 smithi079 ceph-mon[23699]: overall HEALTH_OK 2024-09-21T12:25:00.943 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:00 smithi079 ceph-mon[30023]: mon.c calling monitor election 2024-09-21T12:25:00.943 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:00 smithi079 ceph-mon[30023]: mon.c calling monitor election 2024-09-21T12:25:00.943 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:00 smithi079 ceph-mon[30023]: mon.a calling monitor election 2024-09-21T12:25:00.943 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:00 smithi079 ceph-mon[30023]: mon.b calling monitor election 2024-09-21T12:25:00.943 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:00 smithi079 ceph-mon[30023]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-21T12:25:00.943 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:00 smithi079 ceph-mon[30023]: monmap e3: 3 mons at {a=[v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0],b=[v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0],c=[v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-21T12:25:00.943 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:00 smithi079 ceph-mon[30023]: fsmap 2024-09-21T12:25:00.943 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:00 smithi079 ceph-mon[30023]: osdmap e4: 0 total, 0 up, 0 in 2024-09-21T12:25:00.943 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:00 smithi079 ceph-mon[30023]: mgrmap e13: y(active, since 49s) 2024-09-21T12:25:00.943 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:00 smithi079 ceph-mon[30023]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,b) 2024-09-21T12:25:00.943 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:00 smithi079 ceph-mon[30023]: Cluster is now healthy 2024-09-21T12:25:00.943 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:00 smithi079 ceph-mon[30023]: overall HEALTH_OK 2024-09-21T12:25:02.045 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:25:02.138 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:01 smithi079 ceph-mon[23699]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:25:02.138 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:01 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:02.138 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:01 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T12:25:02.138 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:01 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:02.138 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:01 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:02.138 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:01 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:02.139 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:01 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:02.139 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:01 smithi079 ceph-mon[30023]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:25:02.139 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:01 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:02.139 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:01 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T12:25:02.139 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:01 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:02.139 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:01 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:02.139 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:01 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:02.139 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:01 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:02.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:01 smithi104 ceph-mon[25639]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:25:02.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:01 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:02.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:01 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T12:25:02.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:01 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:02.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:01 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:02.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:01 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:02.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:01 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:04.068 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:03 smithi079 ceph-mon[23699]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:25:04.068 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:03 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:04.068 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:03 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:04.068 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:03 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-21T12:25:04.069 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:03 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-21T12:25:04.069 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:03 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:25:04.069 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:03 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:04.069 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:03 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:04.069 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:03 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:04.069 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:03 smithi079 ceph-mon[30023]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:25:04.069 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:03 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:04.069 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:03 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:04.069 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:03 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-21T12:25:04.069 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:03 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-21T12:25:04.069 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:03 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:25:04.070 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:03 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:04.070 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:03 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:04.070 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:03 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:04.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:03 smithi104 ceph-mon[25639]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:25:04.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:03 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:04.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:03 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:04.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:03 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-21T12:25:04.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:03 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-21T12:25:04.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:03 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:25:04.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:03 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:04.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:03 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:04.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:03 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:04.690 INFO:teuthology.orchestra.run.smithi079.stdout:# minimal ceph.conf for 33168022-7814-11ef-baf6-efdc52797490 2024-09-21T12:25:04.690 INFO:teuthology.orchestra.run.smithi079.stdout:[global] 2024-09-21T12:25:04.690 INFO:teuthology.orchestra.run.smithi079.stdout: fsid = 33168022-7814-11ef-baf6-efdc52797490 2024-09-21T12:25:04.690 INFO:teuthology.orchestra.run.smithi079.stdout: mon_host = [v2:172.21.15.79:3300/0,v1:172.21.15.79:6789/0] [v2:172.21.15.104:3300/0,v1:172.21.15.104:6789/0] [v2:172.21.15.79:3301/0,v1:172.21.15.79:6790/0] 2024-09-21T12:25:04.942 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:04 smithi079 ceph-mon[30023]: Reconfiguring mon.c (monmap changed)... 2024-09-21T12:25:04.942 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:04 smithi079 ceph-mon[30023]: Reconfiguring daemon mon.c on smithi079 2024-09-21T12:25:04.942 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:04 smithi079 ceph-mon[30023]: from='client.? 172.21.15.79:0/1586864879' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:25:04.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:04 smithi079 ceph-mon[23699]: Reconfiguring mon.c (monmap changed)... 2024-09-21T12:25:04.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:04 smithi079 ceph-mon[23699]: Reconfiguring daemon mon.c on smithi079 2024-09-21T12:25:04.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:04 smithi079 ceph-mon[23699]: from='client.? 172.21.15.79:0/1586864879' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:25:05.176 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:04 smithi104 ceph-mon[25639]: Reconfiguring mon.c (monmap changed)... 2024-09-21T12:25:05.176 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:04 smithi104 ceph-mon[25639]: Reconfiguring daemon mon.c on smithi079 2024-09-21T12:25:05.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:04 smithi104 ceph-mon[25639]: from='client.? 172.21.15.79:0/1586864879' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:25:05.428 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-09-21T12:25:05.429 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-09-21T12:25:05.429 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd of=/etc/ceph/ceph.conf 2024-09-21T12:25:05.456 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-09-21T12:25:05.456 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-21T12:25:05.520 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-09-21T12:25:05.520 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/etc/ceph/ceph.conf 2024-09-21T12:25:05.547 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-09-21T12:25:05.547 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-21T12:25:05.613 INFO:tasks.cephadm:Adding mgr.y on smithi079 2024-09-21T12:25:05.613 INFO:tasks.cephadm:Adding mgr.x on smithi104 2024-09-21T12:25:05.613 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 33168022-7814-11ef-baf6-efdc52797490 -- ceph orch apply mgr '2;smithi079=y;smithi104=x' 2024-09-21T12:25:05.918 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:25:05.926 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:05 smithi104 ceph-mon[25639]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:25:05.926 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:05 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:05.926 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:05 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:05.926 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:05 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-21T12:25:05.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:05 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-21T12:25:05.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:05 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:25:06.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:05 smithi079 ceph-mon[30023]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:25:06.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:05 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:06.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:05 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:06.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:05 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-21T12:25:06.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:05 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-21T12:25:06.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:05 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:25:06.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:05 smithi079 ceph-mon[23699]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:25:06.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:05 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:06.193 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:05 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:06.193 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:05 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-21T12:25:06.193 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:05 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-21T12:25:06.193 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:05 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:25:07.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:06 smithi104 ceph-mon[25639]: Reconfiguring mon.b (monmap changed)... 2024-09-21T12:25:07.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:06 smithi104 ceph-mon[25639]: Reconfiguring daemon mon.b on smithi104 2024-09-21T12:25:07.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:06 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:07.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:06 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:07.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:06 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:25:07.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:06 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:25:07.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:06 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:25:07.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:06 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:07.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:06 smithi079 ceph-mon[23699]: Reconfiguring mon.b (monmap changed)... 2024-09-21T12:25:07.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:06 smithi079 ceph-mon[23699]: Reconfiguring daemon mon.b on smithi104 2024-09-21T12:25:07.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:06 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:07.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:06 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:07.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:06 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:25:07.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:06 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:25:07.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:06 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:25:07.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:06 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:07.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:06 smithi079 ceph-mon[30023]: Reconfiguring mon.b (monmap changed)... 2024-09-21T12:25:07.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:06 smithi079 ceph-mon[30023]: Reconfiguring daemon mon.b on smithi104 2024-09-21T12:25:07.193 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:06 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:07.193 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:06 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:07.193 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:06 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:25:07.193 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:06 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:25:07.193 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:06 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:25:07.193 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:06 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:07.940 INFO:teuthology.orchestra.run.smithi104.stdout:Scheduled mgr update... 2024-09-21T12:25:08.176 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:07 smithi104 ceph-mon[25639]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:25:08.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:07 smithi079 ceph-mon[23699]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:25:08.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:07 smithi079 ceph-mon[30023]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:25:08.460 DEBUG:teuthology.orchestra.run.smithi104:mgr.x> sudo journalctl -f -n 0 -u ceph-33168022-7814-11ef-baf6-efdc52797490@mgr.x.service 2024-09-21T12:25:08.462 DEBUG:tasks.cephadm:set 0 configs 2024-09-21T12:25:08.462 INFO:tasks.cephadm:Deploying OSDs... 2024-09-21T12:25:08.462 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-09-21T12:25:08.463 DEBUG:teuthology.orchestra.run.smithi079:> dd if=/scratch_devs of=/dev/stdout 2024-09-21T12:25:08.482 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-21T12:25:08.482 DEBUG:teuthology.orchestra.run.smithi079:> stat /dev/vg_nvme/lv_1 2024-09-21T12:25:08.541 INFO:teuthology.orchestra.run.smithi079.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-21T12:25:08.541 INFO:teuthology.orchestra.run.smithi079.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T12:25:08.541 INFO:teuthology.orchestra.run.smithi079.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-09-21T12:25:08.541 INFO:teuthology.orchestra.run.smithi079.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T12:25:08.541 INFO:teuthology.orchestra.run.smithi079.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T12:25:08.542 INFO:teuthology.orchestra.run.smithi079.stdout:Access: 2024-09-21 12:25:00.256597635 +0000 2024-09-21T12:25:08.542 INFO:teuthology.orchestra.run.smithi079.stdout:Modify: 2024-09-21 12:22:22.082279829 +0000 2024-09-21T12:25:08.542 INFO:teuthology.orchestra.run.smithi079.stdout:Change: 2024-09-21 12:22:22.082279829 +0000 2024-09-21T12:25:08.542 INFO:teuthology.orchestra.run.smithi079.stdout: Birth: 2024-09-21 12:22:22.082279829 +0000 2024-09-21T12:25:08.542 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-21T12:25:08.608 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records in 2024-09-21T12:25:08.608 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records out 2024-09-21T12:25:08.609 INFO:teuthology.orchestra.run.smithi079.stderr:512 bytes copied, 0.000209213 s, 2.4 MB/s 2024-09-21T12:25:08.610 DEBUG:teuthology.orchestra.run.smithi079:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-21T12:25:08.669 DEBUG:teuthology.orchestra.run.smithi079:> stat /dev/vg_nvme/lv_2 2024-09-21T12:25:08.726 INFO:teuthology.orchestra.run.smithi079.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-21T12:25:08.727 INFO:teuthology.orchestra.run.smithi079.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T12:25:08.727 INFO:teuthology.orchestra.run.smithi079.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-09-21T12:25:08.727 INFO:teuthology.orchestra.run.smithi079.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T12:25:08.727 INFO:teuthology.orchestra.run.smithi079.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T12:25:08.727 INFO:teuthology.orchestra.run.smithi079.stdout:Access: 2024-09-21 12:25:00.256597635 +0000 2024-09-21T12:25:08.727 INFO:teuthology.orchestra.run.smithi079.stdout:Modify: 2024-09-21 12:22:22.407272421 +0000 2024-09-21T12:25:08.727 INFO:teuthology.orchestra.run.smithi079.stdout:Change: 2024-09-21 12:22:22.407272421 +0000 2024-09-21T12:25:08.727 INFO:teuthology.orchestra.run.smithi079.stdout: Birth: 2024-09-21 12:22:22.407272421 +0000 2024-09-21T12:25:08.728 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-21T12:25:08.794 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records in 2024-09-21T12:25:08.794 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records out 2024-09-21T12:25:08.794 INFO:teuthology.orchestra.run.smithi079.stderr:512 bytes copied, 0.000214871 s, 2.4 MB/s 2024-09-21T12:25:08.796 DEBUG:teuthology.orchestra.run.smithi079:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-21T12:25:08.855 DEBUG:teuthology.orchestra.run.smithi079:> stat /dev/vg_nvme/lv_3 2024-09-21T12:25:08.912 INFO:teuthology.orchestra.run.smithi079.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-21T12:25:08.912 INFO:teuthology.orchestra.run.smithi079.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T12:25:08.912 INFO:teuthology.orchestra.run.smithi079.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-09-21T12:25:08.912 INFO:teuthology.orchestra.run.smithi079.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T12:25:08.912 INFO:teuthology.orchestra.run.smithi079.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T12:25:08.912 INFO:teuthology.orchestra.run.smithi079.stdout:Access: 2024-09-21 12:25:00.256597635 +0000 2024-09-21T12:25:08.913 INFO:teuthology.orchestra.run.smithi079.stdout:Modify: 2024-09-21 12:22:22.743264763 +0000 2024-09-21T12:25:08.913 INFO:teuthology.orchestra.run.smithi079.stdout:Change: 2024-09-21 12:22:22.743264763 +0000 2024-09-21T12:25:08.913 INFO:teuthology.orchestra.run.smithi079.stdout: Birth: 2024-09-21 12:22:22.743264763 +0000 2024-09-21T12:25:08.913 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-21T12:25:08.977 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records in 2024-09-21T12:25:08.978 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records out 2024-09-21T12:25:08.978 INFO:teuthology.orchestra.run.smithi079.stderr:512 bytes copied, 0.000204763 s, 2.5 MB/s 2024-09-21T12:25:08.979 DEBUG:teuthology.orchestra.run.smithi079:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-21T12:25:09.035 DEBUG:teuthology.orchestra.run.smithi079:> stat /dev/vg_nvme/lv_4 2024-09-21T12:25:09.091 INFO:teuthology.orchestra.run.smithi079.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-21T12:25:09.091 INFO:teuthology.orchestra.run.smithi079.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T12:25:09.091 INFO:teuthology.orchestra.run.smithi079.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-09-21T12:25:09.091 INFO:teuthology.orchestra.run.smithi079.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T12:25:09.091 INFO:teuthology.orchestra.run.smithi079.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T12:25:09.091 INFO:teuthology.orchestra.run.smithi079.stdout:Access: 2024-09-21 12:25:00.256597635 +0000 2024-09-21T12:25:09.091 INFO:teuthology.orchestra.run.smithi079.stdout:Modify: 2024-09-21 12:22:23.064257447 +0000 2024-09-21T12:25:09.091 INFO:teuthology.orchestra.run.smithi079.stdout:Change: 2024-09-21 12:22:23.064257447 +0000 2024-09-21T12:25:09.092 INFO:teuthology.orchestra.run.smithi079.stdout: Birth: 2024-09-21 12:22:23.064257447 +0000 2024-09-21T12:25:09.092 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-21T12:25:09.155 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records in 2024-09-21T12:25:09.155 INFO:teuthology.orchestra.run.smithi079.stderr:1+0 records out 2024-09-21T12:25:09.155 INFO:teuthology.orchestra.run.smithi079.stderr:512 bytes copied, 0.000220006 s, 2.3 MB/s 2024-09-21T12:25:09.156 DEBUG:teuthology.orchestra.run.smithi079:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-21T12:25:09.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:08 smithi079 ceph-mon[23699]: from='client.24106 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi079=y;smithi104=x", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:25:09.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:08 smithi079 ceph-mon[23699]: Saving service mgr spec with placement smithi079=y;smithi104=x;count:2 2024-09-21T12:25:09.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:08 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:09.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:08 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:25:09.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:08 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:25:09.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:08 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:25:09.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:08 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:09.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:08 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-21T12:25:09.193 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:08 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-21T12:25:09.193 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:08 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-21T12:25:09.193 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:08 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:25:09.193 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:08 smithi079 ceph-mon[23699]: Deploying daemon mgr.x on smithi104 2024-09-21T12:25:09.193 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:08 smithi079 ceph-mon[30023]: from='client.24106 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi079=y;smithi104=x", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:25:09.193 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:08 smithi079 ceph-mon[30023]: Saving service mgr spec with placement smithi079=y;smithi104=x;count:2 2024-09-21T12:25:09.193 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:08 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:09.193 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:08 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:25:09.193 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:08 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:25:09.193 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:08 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:25:09.193 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:08 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:09.193 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:08 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-21T12:25:09.194 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:08 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-21T12:25:09.194 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:08 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-21T12:25:09.194 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:08 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:25:09.194 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:08 smithi079 ceph-mon[30023]: Deploying daemon mgr.x on smithi104 2024-09-21T12:25:09.206 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:08 smithi104 ceph-mon[25639]: from='client.24106 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi079=y;smithi104=x", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:25:09.206 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:08 smithi104 ceph-mon[25639]: Saving service mgr spec with placement smithi079=y;smithi104=x;count:2 2024-09-21T12:25:09.206 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:08 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:09.206 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:08 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:25:09.206 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:08 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:25:09.206 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:08 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:25:09.206 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:08 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:09.206 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:08 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-21T12:25:09.206 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:08 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-21T12:25:09.206 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:08 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-21T12:25:09.206 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:08 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:25:09.207 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:08 smithi104 ceph-mon[25639]: Deploying daemon mgr.x on smithi104 2024-09-21T12:25:09.209 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-09-21T12:25:09.209 DEBUG:teuthology.orchestra.run.smithi104:> dd if=/scratch_devs of=/dev/stdout 2024-09-21T12:25:09.225 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-21T12:25:09.225 DEBUG:teuthology.orchestra.run.smithi104:> stat /dev/vg_nvme/lv_1 2024-09-21T12:25:09.281 INFO:teuthology.orchestra.run.smithi104.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-21T12:25:09.281 INFO:teuthology.orchestra.run.smithi104.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T12:25:09.281 INFO:teuthology.orchestra.run.smithi104.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-09-21T12:25:09.281 INFO:teuthology.orchestra.run.smithi104.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T12:25:09.281 INFO:teuthology.orchestra.run.smithi104.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T12:25:09.281 INFO:teuthology.orchestra.run.smithi104.stdout:Access: 2024-09-21 12:25:00.322057579 +0000 2024-09-21T12:25:09.281 INFO:teuthology.orchestra.run.smithi104.stdout:Modify: 2024-09-21 12:22:22.833394062 +0000 2024-09-21T12:25:09.282 INFO:teuthology.orchestra.run.smithi104.stdout:Change: 2024-09-21 12:22:22.833394062 +0000 2024-09-21T12:25:09.282 INFO:teuthology.orchestra.run.smithi104.stdout: Birth: 2024-09-21 12:22:22.833394062 +0000 2024-09-21T12:25:09.282 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-21T12:25:09.346 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records in 2024-09-21T12:25:09.346 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records out 2024-09-21T12:25:09.346 INFO:teuthology.orchestra.run.smithi104.stderr:512 bytes copied, 0.000263828 s, 1.9 MB/s 2024-09-21T12:25:09.347 DEBUG:teuthology.orchestra.run.smithi104:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-21T12:25:09.403 DEBUG:teuthology.orchestra.run.smithi104:> stat /dev/vg_nvme/lv_2 2024-09-21T12:25:09.458 INFO:teuthology.orchestra.run.smithi104.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-21T12:25:09.458 INFO:teuthology.orchestra.run.smithi104.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T12:25:09.458 INFO:teuthology.orchestra.run.smithi104.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-09-21T12:25:09.459 INFO:teuthology.orchestra.run.smithi104.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T12:25:09.459 INFO:teuthology.orchestra.run.smithi104.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T12:25:09.459 INFO:teuthology.orchestra.run.smithi104.stdout:Access: 2024-09-21 12:25:00.322057579 +0000 2024-09-21T12:25:09.459 INFO:teuthology.orchestra.run.smithi104.stdout:Modify: 2024-09-21 12:22:23.159390339 +0000 2024-09-21T12:25:09.459 INFO:teuthology.orchestra.run.smithi104.stdout:Change: 2024-09-21 12:22:23.159390339 +0000 2024-09-21T12:25:09.459 INFO:teuthology.orchestra.run.smithi104.stdout: Birth: 2024-09-21 12:22:23.159390339 +0000 2024-09-21T12:25:09.459 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-21T12:25:09.521 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records in 2024-09-21T12:25:09.522 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records out 2024-09-21T12:25:09.522 INFO:teuthology.orchestra.run.smithi104.stderr:512 bytes copied, 0.000206888 s, 2.5 MB/s 2024-09-21T12:25:09.523 DEBUG:teuthology.orchestra.run.smithi104:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-21T12:25:09.579 DEBUG:teuthology.orchestra.run.smithi104:> stat /dev/vg_nvme/lv_3 2024-09-21T12:25:09.634 INFO:teuthology.orchestra.run.smithi104.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-21T12:25:09.634 INFO:teuthology.orchestra.run.smithi104.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T12:25:09.634 INFO:teuthology.orchestra.run.smithi104.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-09-21T12:25:09.634 INFO:teuthology.orchestra.run.smithi104.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T12:25:09.634 INFO:teuthology.orchestra.run.smithi104.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T12:25:09.635 INFO:teuthology.orchestra.run.smithi104.stdout:Access: 2024-09-21 12:25:00.322057579 +0000 2024-09-21T12:25:09.635 INFO:teuthology.orchestra.run.smithi104.stdout:Modify: 2024-09-21 12:22:23.503386410 +0000 2024-09-21T12:25:09.635 INFO:teuthology.orchestra.run.smithi104.stdout:Change: 2024-09-21 12:22:23.503386410 +0000 2024-09-21T12:25:09.635 INFO:teuthology.orchestra.run.smithi104.stdout: Birth: 2024-09-21 12:22:23.503386410 +0000 2024-09-21T12:25:09.635 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-21T12:25:09.699 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records in 2024-09-21T12:25:09.699 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records out 2024-09-21T12:25:09.699 INFO:teuthology.orchestra.run.smithi104.stderr:512 bytes copied, 0.000208059 s, 2.5 MB/s 2024-09-21T12:25:09.700 DEBUG:teuthology.orchestra.run.smithi104:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-21T12:25:09.756 DEBUG:teuthology.orchestra.run.smithi104:> stat /dev/vg_nvme/lv_4 2024-09-21T12:25:09.814 INFO:teuthology.orchestra.run.smithi104.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-21T12:25:09.814 INFO:teuthology.orchestra.run.smithi104.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T12:25:09.814 INFO:teuthology.orchestra.run.smithi104.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-09-21T12:25:09.814 INFO:teuthology.orchestra.run.smithi104.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T12:25:09.814 INFO:teuthology.orchestra.run.smithi104.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T12:25:09.814 INFO:teuthology.orchestra.run.smithi104.stdout:Access: 2024-09-21 12:25:00.322057579 +0000 2024-09-21T12:25:09.814 INFO:teuthology.orchestra.run.smithi104.stdout:Modify: 2024-09-21 12:22:23.867382253 +0000 2024-09-21T12:25:09.814 INFO:teuthology.orchestra.run.smithi104.stdout:Change: 2024-09-21 12:22:23.867382253 +0000 2024-09-21T12:25:09.815 INFO:teuthology.orchestra.run.smithi104.stdout: Birth: 2024-09-21 12:22:23.867382253 +0000 2024-09-21T12:25:09.815 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-21T12:25:09.937 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records in 2024-09-21T12:25:09.937 INFO:teuthology.orchestra.run.smithi104.stderr:1+0 records out 2024-09-21T12:25:09.937 INFO:teuthology.orchestra.run.smithi104.stderr:512 bytes copied, 0.000220689 s, 2.3 MB/s 2024-09-21T12:25:09.938 DEBUG:teuthology.orchestra.run.smithi104:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-21T12:25:09.954 INFO:tasks.cephadm:Deploying osd.0 on smithi079 with /dev/vg_nvme/lv_4... 2024-09-21T12:25:09.954 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 33168022-7814-11ef-baf6-efdc52797490 -- lvm zap /dev/vg_nvme/lv_4 2024-09-21T12:25:10.027 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:09 smithi104 ceph-mon[25639]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:25:10.151 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:25:10.272 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:09 smithi079 ceph-mon[23699]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:25:10.272 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:09 smithi079 ceph-mon[30023]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:25:10.304 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:25:10 smithi104 systemd[1]: Starting Ceph mgr.x for 33168022-7814-11ef-baf6-efdc52797490... 2024-09-21T12:25:10.598 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:25:10 smithi104 podman[27813]: 2024-09-21 12:25:10.303754383 +0000 UTC m=+0.088405068 container create b0c77ff77579ecb197d757263ec61d6da3219696e0ce3597661f921fe41406ae (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=reef-6a1bdf8, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-09-21T12:25:10.599 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:25:10 smithi104 podman[27813]: 2024-09-21 12:25:10.230683567 +0000 UTC m=+0.015334259 image pull 0adab276c97e6f3143ffbca5eeff37fcd7dc3b5c0f339a9f747381d9993602de quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6 2024-09-21T12:25:10.599 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:25:10 smithi104 podman[27813]: 2024-09-21 12:25:10.523010137 +0000 UTC m=+0.307660828 container init b0c77ff77579ecb197d757263ec61d6da3219696e0ce3597661f921fe41406ae (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=reef-6a1bdf8, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-09-21T12:25:10.599 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:25:10 smithi104 podman[27813]: 2024-09-21 12:25:10.526009881 +0000 UTC m=+0.310660564 container start b0c77ff77579ecb197d757263ec61d6da3219696e0ce3597661f921fe41406ae (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x, RELEASE=reef-6a1bdf8, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2) 2024-09-21T12:25:10.599 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:25:10 smithi104 ceph-mgr[27832]: -- 172.21.15.104:0/3361804560 <== mon.1 v2:172.21.15.104:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55a888fc0d00 con 0x55a888f95400 2024-09-21T12:25:10.599 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:25:10 smithi104 bash[27813]: b0c77ff77579ecb197d757263ec61d6da3219696e0ce3597661f921fe41406ae 2024-09-21T12:25:10.865 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:25:10 smithi104 systemd[1]: Started Ceph mgr.x for 33168022-7814-11ef-baf6-efdc52797490. 2024-09-21T12:25:10.865 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:25:10 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[27828]: 2024-09-21T12:25:10.651+0000 7f593495d040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-21T12:25:10.865 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:25:10 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[27828]: 2024-09-21T12:25:10.709+0000 7f593495d040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-21T12:25:10.865 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:25:10 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[27828]: 2024-09-21T12:25:10.862+0000 7f593495d040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-21T12:25:11.176 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:25:10 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[27828]: 2024-09-21T12:25:10.927+0000 7f593495d040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-21T12:25:11.176 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:25:11 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[27828]: 2024-09-21T12:25:11.070+0000 7f593495d040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-21T12:25:11.526 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:25:11 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[27828]: 2024-09-21T12:25:11.220+0000 7f593495d040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-21T12:25:11.783 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:25:11 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[27828]: 2024-09-21T12:25:11.535+0000 7f593495d040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-21T12:25:11.783 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:25:11 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[27828]: 2024-09-21T12:25:11.597+0000 7f593495d040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-21T12:25:11.783 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:11 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:11.783 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:11 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:11.783 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:11 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:11.783 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:11 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:11.783 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:11 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:25:11.783 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:11 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:25:11.783 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:11 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:25:11.784 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:11 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:11.784 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:11 smithi104 ceph-mon[25639]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-21T12:25:11.784 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:11 smithi104 ceph-mon[25639]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:25:11.784 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:11 smithi104 ceph-mon[25639]: Reconfiguring mgr.y (unknown last config time)... 2024-09-21T12:25:11.784 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:11 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-21T12:25:11.784 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:11 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-21T12:25:11.784 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:11 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:25:11.784 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:11 smithi104 ceph-mon[25639]: Reconfiguring daemon mgr.y on smithi079 2024-09-21T12:25:11.784 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:11 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:11.791 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:11 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:11.791 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:11 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:11.791 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:11 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:11.791 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:11 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:11.791 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:11 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:25:11.791 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:11 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:25:11.792 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:11 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:25:11.792 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:11 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:11.792 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:11 smithi079 ceph-mon[30023]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-21T12:25:11.792 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:11 smithi079 ceph-mon[30023]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:25:11.792 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:11 smithi079 ceph-mon[30023]: Reconfiguring mgr.y (unknown last config time)... 2024-09-21T12:25:11.792 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:11 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-21T12:25:11.792 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:11 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-21T12:25:11.792 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:11 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:25:11.792 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:11 smithi079 ceph-mon[30023]: Reconfiguring daemon mgr.y on smithi079 2024-09-21T12:25:11.792 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:11 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:11.792 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:11 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:11.792 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:11 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:11.792 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:11 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:11.793 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:11 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:11.793 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:11 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:25:11.793 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:11 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:25:11.793 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:11 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:25:11.793 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:11 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:11.793 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:11 smithi079 ceph-mon[23699]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-21T12:25:11.793 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:11 smithi079 ceph-mon[23699]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:25:11.793 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:11 smithi079 ceph-mon[23699]: Reconfiguring mgr.y (unknown last config time)... 2024-09-21T12:25:11.793 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:11 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-21T12:25:11.793 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:11 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-21T12:25:11.793 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:11 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:25:11.793 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:11 smithi079 ceph-mon[23699]: Reconfiguring daemon mgr.y on smithi079 2024-09-21T12:25:11.793 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:11 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:12.176 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:25:11 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[27828]: 2024-09-21T12:25:11.780+0000 7f593495d040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-21T12:25:12.358 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-21T12:25:12.376 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 33168022-7814-11ef-baf6-efdc52797490 -- ceph orch daemon add osd smithi079:vg_nvme/lv_4 2024-09-21T12:25:12.804 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:25:12 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[27828]: 2024-09-21T12:25:12.680+0000 7f593495d040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-21T12:25:12.804 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:25:12 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[27828]: 2024-09-21T12:25:12.748+0000 7f593495d040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-21T12:25:13.489 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:25:13.676 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:25:13 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[27828]: 2024-09-21T12:25:13.516+0000 7f593495d040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-21T12:25:13.677 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:25:13 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[27828]: 2024-09-21T12:25:13.632+0000 7f593495d040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-21T12:25:13.906 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:13 smithi079 ceph-mon[23699]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:25:13.906 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:13 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:13.906 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:13 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:13.906 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:13 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:25:13.906 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:13 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:25:13.906 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:13 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:25:13.906 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:13 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:13.907 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:13 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:13.907 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:13 smithi079 ceph-mon[30023]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:25:13.907 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:13 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:13.907 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:13 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:13.907 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:13 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:25:13.907 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:13 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:25:13.907 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:13 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:25:13.907 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:13 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:13.907 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:13 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:13.962 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:25:13 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[27828]: 2024-09-21T12:25:13.798+0000 7f593495d040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-21T12:25:13.962 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:25:13 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[27828]: 2024-09-21T12:25:13.861+0000 7f593495d040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-21T12:25:13.962 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:13 smithi104 ceph-mon[25639]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:25:13.962 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:13 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:13.962 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:13 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:13.963 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:13 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:25:13.963 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:13 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:25:13.963 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:13 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:25:13.963 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:13 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:13.963 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:13 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:14.304 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:25:14 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[27828]: 2024-09-21T12:25:14.010+0000 7f593495d040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-21T12:25:14.304 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:25:14 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[27828]: 2024-09-21T12:25:14.082+0000 7f593495d040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-21T12:25:14.304 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:25:14 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[27828]: 2024-09-21T12:25:14.142+0000 7f593495d040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-21T12:25:14.304 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:25:14 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[27828]: 2024-09-21T12:25:14.301+0000 7f593495d040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-21T12:25:14.558 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:25:14 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[27828]: 2024-09-21T12:25:14.366+0000 7f593495d040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-21T12:25:14.558 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:25:14 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[27828]: 2024-09-21T12:25:14.441+0000 7f593495d040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-21T12:25:14.832 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:25:14 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[27828]: 2024-09-21T12:25:14.555+0000 7f593495d040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-21T12:25:14.832 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:25:14 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[27828]: 2024-09-21T12:25:14.617+0000 7f593495d040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-21T12:25:14.832 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:25:14 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[27828]: 2024-09-21T12:25:14.678+0000 7f593495d040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-21T12:25:14.889 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:14 smithi079 ceph-mon[23699]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-21T12:25:14.889 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:14 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:14.889 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:14 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:14.889 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:14 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:25:14.889 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:14 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:25:14.889 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:14 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:25:14.889 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:14 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:14.889 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:14 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:14.890 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:14 smithi079 ceph-mon[30023]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-21T12:25:14.890 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:14 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:14.890 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:14 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:14.890 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:14 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:25:14.890 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:14 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:25:14.890 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:14 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:25:14.890 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:14 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:14.890 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:14 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:15.176 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:14 smithi104 ceph-mon[25639]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-21T12:25:15.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:14 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:15.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:14 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:15.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:14 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:25:15.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:14 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:25:15.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:14 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:25:15.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:14 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:15.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:14 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:15.836 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:25:15 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[27828]: 2024-09-21T12:25:15.448+0000 7f593495d040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-21T12:25:16.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:15 smithi104 ceph-mon[25639]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:25:16.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:15 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-21T12:25:16.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:15 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-21T12:25:16.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:15 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:25:16.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:15 smithi104 ceph-mon[25639]: Standby manager daemon x started 2024-09-21T12:25:16.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:15 smithi104 ceph-mon[25639]: from='mgr.? 172.21.15.104:0/3635927393' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-21T12:25:16.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:15 smithi104 ceph-mon[25639]: from='mgr.? 172.21.15.104:0/3635927393' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-21T12:25:16.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:15 smithi104 ceph-mon[25639]: from='mgr.? 172.21.15.104:0/3635927393' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-21T12:25:16.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:15 smithi104 ceph-mon[25639]: from='mgr.? 172.21.15.104:0/3635927393' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-21T12:25:16.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:15 smithi079 ceph-mon[23699]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:25:16.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:15 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-21T12:25:16.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:15 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-21T12:25:16.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:15 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:25:16.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:15 smithi079 ceph-mon[23699]: Standby manager daemon x started 2024-09-21T12:25:16.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:15 smithi079 ceph-mon[23699]: from='mgr.? 172.21.15.104:0/3635927393' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-21T12:25:16.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:15 smithi079 ceph-mon[23699]: from='mgr.? 172.21.15.104:0/3635927393' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-21T12:25:16.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:15 smithi079 ceph-mon[23699]: from='mgr.? 172.21.15.104:0/3635927393' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-21T12:25:16.193 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:15 smithi079 ceph-mon[23699]: from='mgr.? 172.21.15.104:0/3635927393' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-21T12:25:16.193 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:15 smithi079 ceph-mon[30023]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:25:16.193 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:15 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-21T12:25:16.193 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:15 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-21T12:25:16.193 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:15 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:25:16.193 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:15 smithi079 ceph-mon[30023]: Standby manager daemon x started 2024-09-21T12:25:16.193 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:15 smithi079 ceph-mon[30023]: from='mgr.? 172.21.15.104:0/3635927393' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-21T12:25:16.193 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:15 smithi079 ceph-mon[30023]: from='mgr.? 172.21.15.104:0/3635927393' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-21T12:25:16.193 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:15 smithi079 ceph-mon[30023]: from='mgr.? 172.21.15.104:0/3635927393' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-21T12:25:16.193 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:15 smithi079 ceph-mon[30023]: from='mgr.? 172.21.15.104:0/3635927393' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-21T12:25:16.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:16 smithi079 ceph-mon[23699]: from='client.14208 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi079:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:25:16.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:16 smithi079 ceph-mon[23699]: mgrmap e14: y(active, since 65s), standbys: x 2024-09-21T12:25:16.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:16 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-21T12:25:16.942 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:16 smithi079 ceph-mon[30023]: from='client.14208 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi079:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:25:16.942 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:16 smithi079 ceph-mon[30023]: mgrmap e14: y(active, since 65s), standbys: x 2024-09-21T12:25:16.942 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:16 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-21T12:25:17.176 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:16 smithi104 ceph-mon[25639]: from='client.14208 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi079:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:25:17.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:16 smithi104 ceph-mon[25639]: mgrmap e14: y(active, since 65s), standbys: x 2024-09-21T12:25:17.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:16 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-21T12:25:18.167 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:17 smithi079 ceph-mon[23699]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:25:18.168 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:17 smithi079 ceph-mon[23699]: from='client.? 172.21.15.79:0/4268773886' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aa2c4408-1b95-4d23-9f60-d73b204290db"}]: dispatch 2024-09-21T12:25:18.168 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:17 smithi079 ceph-mon[23699]: from='client.? 172.21.15.79:0/4268773886' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "aa2c4408-1b95-4d23-9f60-d73b204290db"}]': finished 2024-09-21T12:25:18.168 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:17 smithi079 ceph-mon[23699]: osdmap e5: 1 total, 0 up, 1 in 2024-09-21T12:25:18.168 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:17 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T12:25:18.168 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:17 smithi079 ceph-mon[30023]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:25:18.168 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:17 smithi079 ceph-mon[30023]: from='client.? 172.21.15.79:0/4268773886' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aa2c4408-1b95-4d23-9f60-d73b204290db"}]: dispatch 2024-09-21T12:25:18.168 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:17 smithi079 ceph-mon[30023]: from='client.? 172.21.15.79:0/4268773886' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "aa2c4408-1b95-4d23-9f60-d73b204290db"}]': finished 2024-09-21T12:25:18.168 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:17 smithi079 ceph-mon[30023]: osdmap e5: 1 total, 0 up, 1 in 2024-09-21T12:25:18.168 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:17 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T12:25:18.176 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:17 smithi104 ceph-mon[25639]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:25:18.176 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:17 smithi104 ceph-mon[25639]: from='client.? 172.21.15.79:0/4268773886' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aa2c4408-1b95-4d23-9f60-d73b204290db"}]: dispatch 2024-09-21T12:25:18.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:17 smithi104 ceph-mon[25639]: from='client.? 172.21.15.79:0/4268773886' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "aa2c4408-1b95-4d23-9f60-d73b204290db"}]': finished 2024-09-21T12:25:18.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:17 smithi104 ceph-mon[25639]: osdmap e5: 1 total, 0 up, 1 in 2024-09-21T12:25:18.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:17 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T12:25:19.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:18 smithi104 ceph-mon[25639]: from='client.? 172.21.15.79:0/2491048811' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-21T12:25:19.170 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:18 smithi079 ceph-mon[23699]: from='client.? 172.21.15.79:0/2491048811' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-21T12:25:19.171 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:18 smithi079 ceph-mon[30023]: from='client.? 172.21.15.79:0/2491048811' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-21T12:25:20.105 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:19 smithi079 ceph-mon[23699]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:25:20.106 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:19 smithi079 ceph-mon[30023]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:25:20.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:19 smithi104 ceph-mon[25639]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:25:22.176 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:21 smithi104 ceph-mon[25639]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:25:22.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:21 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:22.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:21 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:22.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:21 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:22.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:21 smithi079 ceph-mon[23699]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:25:22.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:21 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:22.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:21 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:22.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:21 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:22.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:21 smithi079 ceph-mon[30023]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:25:22.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:21 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:22.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:21 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:22.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:21 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:24.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:23 smithi104 ceph-mon[25639]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:25:24.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:23 smithi079 ceph-mon[23699]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:25:24.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:23 smithi079 ceph-mon[30023]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:25:24.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:24 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-09-21T12:25:24.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:24 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:25:24.942 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:24 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-09-21T12:25:24.942 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:24 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:25:25.176 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:24 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-09-21T12:25:25.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:24 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:25:26.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:25 smithi104 ceph-mon[25639]: Deploying daemon osd.0 on smithi079 2024-09-21T12:25:26.178 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:25 smithi104 ceph-mon[25639]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:25:26.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:25 smithi079 ceph-mon[23699]: Deploying daemon osd.0 on smithi079 2024-09-21T12:25:26.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:25 smithi079 ceph-mon[23699]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:25:26.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:25 smithi079 ceph-mon[30023]: Deploying daemon osd.0 on smithi079 2024-09-21T12:25:26.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:25 smithi079 ceph-mon[30023]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:25:28.013 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:27 smithi079 ceph-mon[23699]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:25:28.013 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:27 smithi079 ceph-mon[30023]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:25:28.176 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:27 smithi104 ceph-mon[25639]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:25:29.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:28 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:25:29.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:28 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:29.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:28 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:25:29.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:28 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:25:29.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:28 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:29.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:28 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:29.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:28 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:25:29.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:28 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:29.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:28 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:25:29.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:28 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:25:29.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:28 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:29.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:28 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:29.193 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:28 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:25:29.193 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:28 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:29.193 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:28 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:25:29.193 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:28 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:25:29.193 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:28 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:29.193 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:28 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:30.176 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:29 smithi104 ceph-mon[25639]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:25:30.191 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:29 smithi079 ceph-mon[23699]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:25:30.191 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:29 smithi079 ceph-mon[30023]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:25:31.233 INFO:teuthology.orchestra.run.smithi079.stdout:Created osd(s) 0 on host 'smithi079' 2024-09-21T12:25:31.828 DEBUG:teuthology.orchestra.run.smithi079:osd.0> sudo journalctl -f -n 0 -u ceph-33168022-7814-11ef-baf6-efdc52797490@osd.0.service 2024-09-21T12:25:31.831 INFO:tasks.cephadm:Deploying osd.1 on smithi079 with /dev/vg_nvme/lv_3... 2024-09-21T12:25:31.831 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 33168022-7814-11ef-baf6-efdc52797490 -- lvm zap /dev/vg_nvme/lv_3 2024-09-21T12:25:32.065 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:25:32.081 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:31 smithi104 ceph-mon[25639]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:25:32.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:31 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:25:32.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:31 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:32.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:31 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-21T12:25:32.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:31 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:32.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:31 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:32.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:31 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:25:32.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:31 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:25:32.082 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:31 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:32.158 INFO:journalctl@ceph.osd.0.smithi079.stdout:Sep 21 12:25:32 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-0[34281]: 2024-09-21T12:25:32.021+0000 7f31d05bf640 -1 osd.0 0 log_to_monitors true 2024-09-21T12:25:32.158 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:31 smithi079 ceph-mon[23699]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:25:32.158 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:31 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:25:32.158 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:31 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:32.159 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:31 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-21T12:25:32.159 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:31 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:32.159 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:31 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:32.159 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:31 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:25:32.159 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:31 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:25:32.159 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:31 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:32.159 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:31 smithi079 ceph-mon[30023]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:25:32.159 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:31 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:25:32.159 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:31 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:32.159 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:31 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-21T12:25:32.159 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:31 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:32.159 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:31 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:32.160 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:31 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:25:32.160 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:31 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:25:32.160 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:31 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:33.161 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:32 smithi079 ceph-mon[23699]: Adjusting osd_memory_target on smithi079 to 15040M 2024-09-21T12:25:33.161 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:32 smithi079 ceph-mon[23699]: from='osd.0 [v2:172.21.15.79:6802/846480130,v1:172.21.15.79:6803/846480130]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-21T12:25:33.161 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:32 smithi079 ceph-mon[23699]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-21T12:25:33.161 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:32 smithi079 ceph-mon[30023]: Adjusting osd_memory_target on smithi079 to 15040M 2024-09-21T12:25:33.161 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:32 smithi079 ceph-mon[30023]: from='osd.0 [v2:172.21.15.79:6802/846480130,v1:172.21.15.79:6803/846480130]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-21T12:25:33.162 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:32 smithi079 ceph-mon[30023]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-21T12:25:33.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:32 smithi104 ceph-mon[25639]: Adjusting osd_memory_target on smithi079 to 15040M 2024-09-21T12:25:33.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:32 smithi104 ceph-mon[25639]: from='osd.0 [v2:172.21.15.79:6802/846480130,v1:172.21.15.79:6803/846480130]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-21T12:25:33.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:32 smithi104 ceph-mon[25639]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-21T12:25:34.160 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:33 smithi104 ceph-mon[25639]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:25:34.160 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:33 smithi104 ceph-mon[25639]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-21T12:25:34.160 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:33 smithi104 ceph-mon[25639]: osdmap e6: 1 total, 0 up, 1 in 2024-09-21T12:25:34.160 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:33 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T12:25:34.161 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:33 smithi104 ceph-mon[25639]: from='osd.0 [v2:172.21.15.79:6802/846480130,v1:172.21.15.79:6803/846480130]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]: dispatch 2024-09-21T12:25:34.161 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:33 smithi104 ceph-mon[25639]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]: dispatch 2024-09-21T12:25:34.161 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:33 smithi079 ceph-mon[23699]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:25:34.161 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:33 smithi079 ceph-mon[23699]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-21T12:25:34.161 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:33 smithi079 ceph-mon[23699]: osdmap e6: 1 total, 0 up, 1 in 2024-09-21T12:25:34.161 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:33 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T12:25:34.161 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:33 smithi079 ceph-mon[23699]: from='osd.0 [v2:172.21.15.79:6802/846480130,v1:172.21.15.79:6803/846480130]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]: dispatch 2024-09-21T12:25:34.161 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:33 smithi079 ceph-mon[23699]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]: dispatch 2024-09-21T12:25:34.162 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:33 smithi079 ceph-mon[30023]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:25:34.162 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:33 smithi079 ceph-mon[30023]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-21T12:25:34.162 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:33 smithi079 ceph-mon[30023]: osdmap e6: 1 total, 0 up, 1 in 2024-09-21T12:25:34.162 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:33 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T12:25:34.162 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:33 smithi079 ceph-mon[30023]: from='osd.0 [v2:172.21.15.79:6802/846480130,v1:172.21.15.79:6803/846480130]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]: dispatch 2024-09-21T12:25:34.162 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:33 smithi079 ceph-mon[30023]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]: dispatch 2024-09-21T12:25:34.307 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-21T12:25:34.332 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 33168022-7814-11ef-baf6-efdc52797490 -- ceph orch daemon add osd smithi079:vg_nvme/lv_3 2024-09-21T12:25:34.442 INFO:journalctl@ceph.osd.0.smithi079.stdout:Sep 21 12:25:34 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-0[34281]: 2024-09-21T12:25:34.366+0000 7f31cc540640 -1 osd.0 0 waiting for initial osdmap 2024-09-21T12:25:34.442 INFO:journalctl@ceph.osd.0.smithi079.stdout:Sep 21 12:25:34 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-0[34281]: 2024-09-21T12:25:34.370+0000 7f31c4361640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T12:25:34.528 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:25:35.178 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:34 smithi104 ceph-mon[25639]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]': finished 2024-09-21T12:25:35.178 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:34 smithi104 ceph-mon[25639]: osdmap e7: 1 total, 0 up, 1 in 2024-09-21T12:25:35.178 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:34 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T12:25:35.178 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:34 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T12:25:35.178 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:34 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:35.178 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:34 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:35.178 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:34 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:35.178 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:34 smithi104 ceph-mon[25639]: from='osd.0 ' entity='osd.0' 2024-09-21T12:25:35.194 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:34 smithi079 ceph-mon[30023]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]': finished 2024-09-21T12:25:35.194 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:34 smithi079 ceph-mon[30023]: osdmap e7: 1 total, 0 up, 1 in 2024-09-21T12:25:35.194 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:34 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T12:25:35.194 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:34 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T12:25:35.194 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:34 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:35.194 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:34 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:35.194 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:34 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:35.194 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:34 smithi079 ceph-mon[30023]: from='osd.0 ' entity='osd.0' 2024-09-21T12:25:35.195 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:34 smithi079 ceph-mon[23699]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]': finished 2024-09-21T12:25:35.195 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:34 smithi079 ceph-mon[23699]: osdmap e7: 1 total, 0 up, 1 in 2024-09-21T12:25:35.195 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:34 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T12:25:35.195 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:34 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T12:25:35.195 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:34 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:35.195 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:34 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:35.195 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:34 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:35.195 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:34 smithi079 ceph-mon[23699]: from='osd.0 ' entity='osd.0' 2024-09-21T12:25:36.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:35 smithi104 ceph-mon[25639]: purged_snaps scrub starts 2024-09-21T12:25:36.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:35 smithi104 ceph-mon[25639]: purged_snaps scrub ok 2024-09-21T12:25:36.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:35 smithi104 ceph-mon[25639]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:25:36.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:35 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T12:25:36.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:35 smithi104 ceph-mon[25639]: osd.0 [v2:172.21.15.79:6802/846480130,v1:172.21.15.79:6803/846480130] boot 2024-09-21T12:25:36.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:35 smithi104 ceph-mon[25639]: osdmap e8: 1 total, 1 up, 1 in 2024-09-21T12:25:36.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:35 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T12:25:36.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:35 smithi079 ceph-mon[23699]: purged_snaps scrub starts 2024-09-21T12:25:36.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:35 smithi079 ceph-mon[23699]: purged_snaps scrub ok 2024-09-21T12:25:36.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:35 smithi079 ceph-mon[23699]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:25:36.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:35 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T12:25:36.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:35 smithi079 ceph-mon[23699]: osd.0 [v2:172.21.15.79:6802/846480130,v1:172.21.15.79:6803/846480130] boot 2024-09-21T12:25:36.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:35 smithi079 ceph-mon[23699]: osdmap e8: 1 total, 1 up, 1 in 2024-09-21T12:25:36.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:35 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T12:25:36.193 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:35 smithi079 ceph-mon[30023]: purged_snaps scrub starts 2024-09-21T12:25:36.193 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:35 smithi079 ceph-mon[30023]: purged_snaps scrub ok 2024-09-21T12:25:36.193 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:35 smithi079 ceph-mon[30023]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T12:25:36.193 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:35 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T12:25:36.193 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:35 smithi079 ceph-mon[30023]: osd.0 [v2:172.21.15.79:6802/846480130,v1:172.21.15.79:6803/846480130] boot 2024-09-21T12:25:36.193 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:35 smithi079 ceph-mon[30023]: osdmap e8: 1 total, 1 up, 1 in 2024-09-21T12:25:36.193 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:35 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T12:25:37.176 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:36 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-21T12:25:37.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:36 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-21T12:25:37.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:36 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:25:37.191 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:36 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-21T12:25:37.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:36 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-21T12:25:37.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:36 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:25:37.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:36 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-21T12:25:37.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:36 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-21T12:25:37.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:36 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:25:37.988 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:37 smithi079 ceph-mon[23699]: from='client.14235 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi079:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:25:37.988 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:37 smithi079 ceph-mon[23699]: pgmap v41: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T12:25:37.988 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:37 smithi079 ceph-mon[23699]: osdmap e9: 1 total, 1 up, 1 in 2024-09-21T12:25:37.988 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:37 smithi079 ceph-mon[30023]: from='client.14235 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi079:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:25:37.988 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:37 smithi079 ceph-mon[30023]: pgmap v41: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T12:25:37.989 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:37 smithi079 ceph-mon[30023]: osdmap e9: 1 total, 1 up, 1 in 2024-09-21T12:25:38.176 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:37 smithi104 ceph-mon[25639]: from='client.14235 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi079:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:25:38.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:37 smithi104 ceph-mon[25639]: pgmap v41: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T12:25:38.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:37 smithi104 ceph-mon[25639]: osdmap e9: 1 total, 1 up, 1 in 2024-09-21T12:25:39.176 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:38 smithi104 ceph-mon[25639]: from='client.? 172.21.15.79:0/3229010755' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a05b4be6-79b2-4dde-bd51-14cafd833c5a"}]: dispatch 2024-09-21T12:25:39.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:38 smithi104 ceph-mon[25639]: from='client.? 172.21.15.79:0/3229010755' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a05b4be6-79b2-4dde-bd51-14cafd833c5a"}]': finished 2024-09-21T12:25:39.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:38 smithi104 ceph-mon[25639]: osdmap e10: 2 total, 1 up, 2 in 2024-09-21T12:25:39.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:38 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T12:25:39.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:38 smithi104 ceph-mon[25639]: from='client.? 172.21.15.79:0/2444842073' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-21T12:25:39.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:38 smithi079 ceph-mon[30023]: from='client.? 172.21.15.79:0/3229010755' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a05b4be6-79b2-4dde-bd51-14cafd833c5a"}]: dispatch 2024-09-21T12:25:39.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:38 smithi079 ceph-mon[30023]: from='client.? 172.21.15.79:0/3229010755' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a05b4be6-79b2-4dde-bd51-14cafd833c5a"}]': finished 2024-09-21T12:25:39.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:38 smithi079 ceph-mon[30023]: osdmap e10: 2 total, 1 up, 2 in 2024-09-21T12:25:39.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:38 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T12:25:39.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:38 smithi079 ceph-mon[30023]: from='client.? 172.21.15.79:0/2444842073' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-21T12:25:39.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:38 smithi079 ceph-mon[23699]: from='client.? 172.21.15.79:0/3229010755' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a05b4be6-79b2-4dde-bd51-14cafd833c5a"}]: dispatch 2024-09-21T12:25:39.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:38 smithi079 ceph-mon[23699]: from='client.? 172.21.15.79:0/3229010755' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a05b4be6-79b2-4dde-bd51-14cafd833c5a"}]': finished 2024-09-21T12:25:39.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:38 smithi079 ceph-mon[23699]: osdmap e10: 2 total, 1 up, 2 in 2024-09-21T12:25:39.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:38 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T12:25:39.193 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:38 smithi079 ceph-mon[23699]: from='client.? 172.21.15.79:0/2444842073' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-21T12:25:40.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:39 smithi079 ceph-mon[23699]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T12:25:40.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:39 smithi079 ceph-mon[30023]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T12:25:40.426 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:39 smithi104 ceph-mon[25639]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T12:25:42.196 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:42 smithi079 ceph-mon[23699]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T12:25:42.197 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:42 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:42.197 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:42 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:42.197 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:42 smithi079 ceph-mon[23699]: Detected new or changed devices on smithi079 2024-09-21T12:25:42.197 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:42 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:25:42.197 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:42 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:42.197 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:42 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:25:42.197 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:42 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:25:42.197 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:42 smithi079 ceph-mon[23699]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T12:25:42.197 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:42 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:42.197 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:42 smithi079 ceph-mon[30023]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T12:25:42.197 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:42 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:42.198 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:42 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:42.198 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:42 smithi079 ceph-mon[30023]: Detected new or changed devices on smithi079 2024-09-21T12:25:42.198 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:42 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:25:42.198 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:42 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:42.198 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:42 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:25:42.198 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:42 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:25:42.198 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:42 smithi079 ceph-mon[30023]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T12:25:42.198 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:42 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:42.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:42 smithi104 ceph-mon[25639]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T12:25:42.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:42 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:42.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:42 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:42.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:42 smithi104 ceph-mon[25639]: Detected new or changed devices on smithi079 2024-09-21T12:25:42.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:42 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:25:42.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:42 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:42.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:42 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:25:42.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:42 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:25:42.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:42 smithi104 ceph-mon[25639]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T12:25:42.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:42 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:43.332 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:43 smithi079 ceph-mon[23699]: Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON) 2024-09-21T12:25:43.332 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:43 smithi079 ceph-mon[30023]: Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON) 2024-09-21T12:25:43.426 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:43 smithi104 ceph-mon[25639]: Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON) 2024-09-21T12:25:44.352 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:44 smithi079 ceph-mon[23699]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T12:25:44.352 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:44 smithi079 ceph-mon[30023]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T12:25:44.426 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:44 smithi104 ceph-mon[25639]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T12:25:46.023 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:45 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:46.024 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:45 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:46.024 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:45 smithi079 ceph-mon[23699]: Detected new or changed devices on smithi079 2024-09-21T12:25:46.024 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:45 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:25:46.024 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:45 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:46.024 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:45 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:25:46.024 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:45 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:25:46.024 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:45 smithi079 ceph-mon[23699]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T12:25:46.024 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:45 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:46.024 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:45 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-09-21T12:25:46.024 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:45 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:25:46.024 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:45 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:46.025 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:45 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:46.025 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:45 smithi079 ceph-mon[30023]: Detected new or changed devices on smithi079 2024-09-21T12:25:46.025 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:45 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:25:46.025 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:45 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:46.025 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:45 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:25:46.025 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:45 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:25:46.025 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:45 smithi079 ceph-mon[30023]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T12:25:46.025 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:45 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:46.025 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:45 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-09-21T12:25:46.025 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:45 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:25:46.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:45 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:46.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:45 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:46.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:45 smithi104 ceph-mon[25639]: Detected new or changed devices on smithi079 2024-09-21T12:25:46.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:45 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:25:46.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:45 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:46.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:45 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:25:46.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:45 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:25:46.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:45 smithi104 ceph-mon[25639]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T12:25:46.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:45 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:46.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:45 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-09-21T12:25:46.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:45 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:25:47.163 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:46 smithi079 ceph-mon[23699]: Deploying daemon osd.1 on smithi079 2024-09-21T12:25:47.163 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:46 smithi079 ceph-mon[23699]: Health check cleared: CEPHADM_STRAY_DAEMON (was: 1 stray daemon(s) not managed by cephadm) 2024-09-21T12:25:47.163 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:46 smithi079 ceph-mon[23699]: Cluster is now healthy 2024-09-21T12:25:47.163 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:46 smithi079 ceph-mon[30023]: Deploying daemon osd.1 on smithi079 2024-09-21T12:25:47.164 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:46 smithi079 ceph-mon[30023]: Health check cleared: CEPHADM_STRAY_DAEMON (was: 1 stray daemon(s) not managed by cephadm) 2024-09-21T12:25:47.164 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:46 smithi079 ceph-mon[30023]: Cluster is now healthy 2024-09-21T12:25:47.176 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:46 smithi104 ceph-mon[25639]: Deploying daemon osd.1 on smithi079 2024-09-21T12:25:47.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:46 smithi104 ceph-mon[25639]: Health check cleared: CEPHADM_STRAY_DAEMON (was: 1 stray daemon(s) not managed by cephadm) 2024-09-21T12:25:47.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:46 smithi104 ceph-mon[25639]: Cluster is now healthy 2024-09-21T12:25:48.048 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:47 smithi079 ceph-mon[23699]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T12:25:48.049 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:47 smithi079 ceph-mon[30023]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T12:25:48.176 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:47 smithi104 ceph-mon[25639]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T12:25:50.176 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:49 smithi104 ceph-mon[25639]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T12:25:50.191 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:49 smithi079 ceph-mon[23699]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T12:25:50.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:49 smithi079 ceph-mon[30023]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T12:25:51.133 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:50 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:25:51.133 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:50 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:25:51.133 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:50 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:25:51.133 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:50 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:51.133 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:50 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:51.133 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:50 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:51.134 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:50 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:25:51.134 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:50 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:25:51.134 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:50 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:25:51.134 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:50 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:51.134 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:50 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:51.134 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:50 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:51.144 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:50 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:25:51.144 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:50 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:25:51.144 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:50 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:25:51.144 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:50 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:51.145 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:50 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:51.145 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:50 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:51.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:51 smithi079 ceph-mon[23699]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T12:25:51.942 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:51 smithi079 ceph-mon[30023]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T12:25:52.176 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:51 smithi104 ceph-mon[25639]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T12:25:52.686 INFO:teuthology.orchestra.run.smithi079.stdout:Created osd(s) 1 on host 'smithi079' 2024-09-21T12:25:52.926 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:52 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:25:52.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:52 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:52.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:52 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-21T12:25:52.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:52 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-21T12:25:52.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:52 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:52.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:52 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:52.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:52 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:25:52.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:52 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:25:52.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:52 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:52.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:52 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:25:52.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:52 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:52.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:52 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-21T12:25:52.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:52 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-21T12:25:52.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:52 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:52.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:52 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:52.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:52 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:25:52.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:52 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:25:52.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:52 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:52.942 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:52 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:25:52.942 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:52 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:52.943 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:52 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-21T12:25:52.943 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:52 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-21T12:25:52.943 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:52 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:52.943 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:52 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:52.943 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:52 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:25:52.943 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:52 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:25:52.943 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:52 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:53.387 DEBUG:teuthology.orchestra.run.smithi079:osd.1> sudo journalctl -f -n 0 -u ceph-33168022-7814-11ef-baf6-efdc52797490@osd.1.service 2024-09-21T12:25:53.389 INFO:tasks.cephadm:Deploying osd.2 on smithi079 with /dev/vg_nvme/lv_2... 2024-09-21T12:25:53.390 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 33168022-7814-11ef-baf6-efdc52797490 -- lvm zap /dev/vg_nvme/lv_2 2024-09-21T12:25:53.620 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:25:54.021 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:53 smithi079 ceph-mon[23699]: Adjusting osd_memory_target on smithi079 to 7520M 2024-09-21T12:25:54.021 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:53 smithi079 ceph-mon[23699]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T12:25:54.021 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:53 smithi079 ceph-mon[23699]: from='osd.1 [v2:172.21.15.79:6810/1764722078,v1:172.21.15.79:6811/1764722078]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-21T12:25:54.021 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:53 smithi079 ceph-mon[30023]: Adjusting osd_memory_target on smithi079 to 7520M 2024-09-21T12:25:54.021 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:53 smithi079 ceph-mon[30023]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T12:25:54.022 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:53 smithi079 ceph-mon[30023]: from='osd.1 [v2:172.21.15.79:6810/1764722078,v1:172.21.15.79:6811/1764722078]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-21T12:25:54.032 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:53 smithi104 ceph-mon[25639]: Adjusting osd_memory_target on smithi079 to 7520M 2024-09-21T12:25:54.032 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:53 smithi104 ceph-mon[25639]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T12:25:54.032 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:53 smithi104 ceph-mon[25639]: from='osd.1 [v2:172.21.15.79:6810/1764722078,v1:172.21.15.79:6811/1764722078]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-21T12:25:55.060 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:54 smithi079 ceph-mon[30023]: from='osd.1 [v2:172.21.15.79:6810/1764722078,v1:172.21.15.79:6811/1764722078]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-21T12:25:55.060 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:54 smithi079 ceph-mon[30023]: osdmap e11: 2 total, 1 up, 2 in 2024-09-21T12:25:55.060 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:54 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T12:25:55.060 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:54 smithi079 ceph-mon[30023]: from='osd.1 [v2:172.21.15.79:6810/1764722078,v1:172.21.15.79:6811/1764722078]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]: dispatch 2024-09-21T12:25:55.060 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:54 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:55.060 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:54 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:55.060 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:54 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:55.061 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:54 smithi079 ceph-mon[23699]: from='osd.1 [v2:172.21.15.79:6810/1764722078,v1:172.21.15.79:6811/1764722078]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-21T12:25:55.061 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:54 smithi079 ceph-mon[23699]: osdmap e11: 2 total, 1 up, 2 in 2024-09-21T12:25:55.061 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:54 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T12:25:55.061 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:54 smithi079 ceph-mon[23699]: from='osd.1 [v2:172.21.15.79:6810/1764722078,v1:172.21.15.79:6811/1764722078]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]: dispatch 2024-09-21T12:25:55.061 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:54 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:55.061 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:54 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:55.061 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:54 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:55.061 INFO:journalctl@ceph.osd.1.smithi079.stdout:Sep 21 12:25:54 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-1[37753]: 2024-09-21T12:25:54.901+0000 7f26a62ab640 -1 osd.1 0 waiting for initial osdmap 2024-09-21T12:25:55.062 INFO:journalctl@ceph.osd.1.smithi079.stdout:Sep 21 12:25:54 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-1[37753]: 2024-09-21T12:25:54.905+0000 7f26a20d4640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T12:25:55.176 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:54 smithi104 ceph-mon[25639]: from='osd.1 [v2:172.21.15.79:6810/1764722078,v1:172.21.15.79:6811/1764722078]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-21T12:25:55.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:54 smithi104 ceph-mon[25639]: osdmap e11: 2 total, 1 up, 2 in 2024-09-21T12:25:55.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:54 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T12:25:55.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:54 smithi104 ceph-mon[25639]: from='osd.1 [v2:172.21.15.79:6810/1764722078,v1:172.21.15.79:6811/1764722078]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]: dispatch 2024-09-21T12:25:55.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:54 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:55.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:54 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:55.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:54 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:25:55.909 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-21T12:25:55.930 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 33168022-7814-11ef-baf6-efdc52797490 -- ceph orch daemon add osd smithi079:vg_nvme/lv_2 2024-09-21T12:25:55.954 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:55 smithi079 ceph-mon[30023]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T12:25:55.954 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:55 smithi079 ceph-mon[30023]: from='osd.1 [v2:172.21.15.79:6810/1764722078,v1:172.21.15.79:6811/1764722078]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]': finished 2024-09-21T12:25:55.954 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:55 smithi079 ceph-mon[30023]: osdmap e12: 2 total, 1 up, 2 in 2024-09-21T12:25:55.954 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:55 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T12:25:55.954 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:55 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T12:25:55.955 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:55 smithi079 ceph-mon[23699]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T12:25:55.955 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:55 smithi079 ceph-mon[23699]: from='osd.1 [v2:172.21.15.79:6810/1764722078,v1:172.21.15.79:6811/1764722078]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]': finished 2024-09-21T12:25:55.955 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:55 smithi079 ceph-mon[23699]: osdmap e12: 2 total, 1 up, 2 in 2024-09-21T12:25:55.955 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:55 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T12:25:55.955 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:55 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T12:25:56.125 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:25:56.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:55 smithi104 ceph-mon[25639]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T12:25:56.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:55 smithi104 ceph-mon[25639]: from='osd.1 [v2:172.21.15.79:6810/1764722078,v1:172.21.15.79:6811/1764722078]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]': finished 2024-09-21T12:25:56.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:55 smithi104 ceph-mon[25639]: osdmap e12: 2 total, 1 up, 2 in 2024-09-21T12:25:56.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:55 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T12:25:56.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:55 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T12:25:57.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:56 smithi104 ceph-mon[25639]: purged_snaps scrub starts 2024-09-21T12:25:57.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:56 smithi104 ceph-mon[25639]: purged_snaps scrub ok 2024-09-21T12:25:57.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:56 smithi104 ceph-mon[25639]: osd.1 [v2:172.21.15.79:6810/1764722078,v1:172.21.15.79:6811/1764722078] boot 2024-09-21T12:25:57.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:56 smithi104 ceph-mon[25639]: osdmap e13: 2 total, 2 up, 2 in 2024-09-21T12:25:57.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:56 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T12:25:57.191 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:56 smithi079 ceph-mon[23699]: purged_snaps scrub starts 2024-09-21T12:25:57.191 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:56 smithi079 ceph-mon[23699]: purged_snaps scrub ok 2024-09-21T12:25:57.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:56 smithi079 ceph-mon[23699]: osd.1 [v2:172.21.15.79:6810/1764722078,v1:172.21.15.79:6811/1764722078] boot 2024-09-21T12:25:57.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:56 smithi079 ceph-mon[23699]: osdmap e13: 2 total, 2 up, 2 in 2024-09-21T12:25:57.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:56 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T12:25:57.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:56 smithi079 ceph-mon[30023]: purged_snaps scrub starts 2024-09-21T12:25:57.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:56 smithi079 ceph-mon[30023]: purged_snaps scrub ok 2024-09-21T12:25:57.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:56 smithi079 ceph-mon[30023]: osd.1 [v2:172.21.15.79:6810/1764722078,v1:172.21.15.79:6811/1764722078] boot 2024-09-21T12:25:57.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:56 smithi079 ceph-mon[30023]: osdmap e13: 2 total, 2 up, 2 in 2024-09-21T12:25:57.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:56 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T12:25:58.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:57 smithi104 ceph-mon[25639]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T12:25:58.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:57 smithi104 ceph-mon[25639]: osdmap e14: 2 total, 2 up, 2 in 2024-09-21T12:25:58.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:57 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-21T12:25:58.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:57 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-21T12:25:58.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:57 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:25:58.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:57 smithi079 ceph-mon[23699]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T12:25:58.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:57 smithi079 ceph-mon[23699]: osdmap e14: 2 total, 2 up, 2 in 2024-09-21T12:25:58.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:57 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-21T12:25:58.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:57 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-21T12:25:58.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:57 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:25:58.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:57 smithi079 ceph-mon[30023]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T12:25:58.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:57 smithi079 ceph-mon[30023]: osdmap e14: 2 total, 2 up, 2 in 2024-09-21T12:25:58.193 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:57 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-21T12:25:58.193 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:57 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-21T12:25:58.193 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:57 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:25:59.176 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:58 smithi104 ceph-mon[25639]: from='client.14262 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi079:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:25:59.183 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:58 smithi079 ceph-mon[23699]: from='client.14262 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi079:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:25:59.183 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:58 smithi079 ceph-mon[30023]: from='client.14262 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi079:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:26:00.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:59 smithi104 ceph-mon[25639]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T12:26:00.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:59 smithi104 ceph-mon[25639]: from='client.? 172.21.15.79:0/3096573234' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8b9ddbf7-8264-4e90-bcf0-e63a7f5e7e9c"}]: dispatch 2024-09-21T12:26:00.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:59 smithi104 ceph-mon[25639]: from='client.? 172.21.15.79:0/3096573234' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8b9ddbf7-8264-4e90-bcf0-e63a7f5e7e9c"}]': finished 2024-09-21T12:26:00.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:59 smithi104 ceph-mon[25639]: osdmap e15: 3 total, 2 up, 3 in 2024-09-21T12:26:00.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:25:59 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T12:26:00.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:59 smithi079 ceph-mon[30023]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T12:26:00.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:59 smithi079 ceph-mon[30023]: from='client.? 172.21.15.79:0/3096573234' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8b9ddbf7-8264-4e90-bcf0-e63a7f5e7e9c"}]: dispatch 2024-09-21T12:26:00.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:59 smithi079 ceph-mon[30023]: from='client.? 172.21.15.79:0/3096573234' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8b9ddbf7-8264-4e90-bcf0-e63a7f5e7e9c"}]': finished 2024-09-21T12:26:00.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:59 smithi079 ceph-mon[30023]: osdmap e15: 3 total, 2 up, 3 in 2024-09-21T12:26:00.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:25:59 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T12:26:00.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:59 smithi079 ceph-mon[23699]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T12:26:00.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:59 smithi079 ceph-mon[23699]: from='client.? 172.21.15.79:0/3096573234' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8b9ddbf7-8264-4e90-bcf0-e63a7f5e7e9c"}]: dispatch 2024-09-21T12:26:00.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:59 smithi079 ceph-mon[23699]: from='client.? 172.21.15.79:0/3096573234' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8b9ddbf7-8264-4e90-bcf0-e63a7f5e7e9c"}]': finished 2024-09-21T12:26:00.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:59 smithi079 ceph-mon[23699]: osdmap e15: 3 total, 2 up, 3 in 2024-09-21T12:26:00.193 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:25:59 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T12:26:01.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:00 smithi079 ceph-mon[23699]: from='client.? 172.21.15.79:0/2788765048' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-21T12:26:01.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:00 smithi079 ceph-mon[30023]: from='client.? 172.21.15.79:0/2788765048' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-21T12:26:01.426 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:00 smithi104 ceph-mon[25639]: from='client.? 172.21.15.79:0/2788765048' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-21T12:26:02.152 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:01 smithi079 ceph-mon[23699]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T12:26:02.153 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:01 smithi079 ceph-mon[30023]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T12:26:02.426 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:01 smithi104 ceph-mon[25639]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T12:26:04.189 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:03 smithi079 ceph-mon[23699]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T12:26:04.190 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:03 smithi079 ceph-mon[30023]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T12:26:04.426 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:03 smithi104 ceph-mon[25639]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T12:26:06.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:05 smithi079 ceph-mon[30023]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T12:26:06.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:05 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:06.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:05 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:06.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:05 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:26:06.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:05 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:06.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:05 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:26:06.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:05 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:26:06.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:05 smithi079 ceph-mon[30023]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T12:26:06.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:05 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:06.193 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:05 smithi079 ceph-mon[23699]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T12:26:06.193 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:05 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:06.193 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:05 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:06.193 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:05 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:26:06.193 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:05 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:06.193 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:05 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:26:06.193 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:05 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:26:06.193 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:05 smithi079 ceph-mon[23699]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T12:26:06.193 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:05 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:06.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:05 smithi104 ceph-mon[25639]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T12:26:06.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:05 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:06.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:05 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:06.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:05 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:26:06.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:05 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:06.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:05 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:26:06.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:05 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:26:06.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:05 smithi104 ceph-mon[25639]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T12:26:06.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:05 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:07.147 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:06 smithi079 ceph-mon[23699]: Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON) 2024-09-21T12:26:07.147 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:06 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:07.147 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:06 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:07.147 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:06 smithi079 ceph-mon[23699]: Detected new or changed devices on smithi079 2024-09-21T12:26:07.147 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:06 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:26:07.147 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:06 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:07.147 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:06 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:26:07.147 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:06 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:26:07.147 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:06 smithi079 ceph-mon[23699]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T12:26:07.147 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:06 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:07.147 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:06 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-09-21T12:26:07.148 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:06 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:26:07.148 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:06 smithi079 ceph-mon[30023]: Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON) 2024-09-21T12:26:07.148 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:06 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:07.148 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:06 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:07.148 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:06 smithi079 ceph-mon[30023]: Detected new or changed devices on smithi079 2024-09-21T12:26:07.148 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:06 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:26:07.148 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:06 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:07.148 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:06 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:26:07.148 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:06 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:26:07.148 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:06 smithi079 ceph-mon[30023]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T12:26:07.148 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:06 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:07.149 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:06 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-09-21T12:26:07.149 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:06 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:26:07.426 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:06 smithi104 ceph-mon[25639]: Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON) 2024-09-21T12:26:07.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:06 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:07.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:06 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:07.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:06 smithi104 ceph-mon[25639]: Detected new or changed devices on smithi079 2024-09-21T12:26:07.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:06 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:26:07.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:06 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:07.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:06 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:26:07.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:06 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:26:07.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:06 smithi104 ceph-mon[25639]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T12:26:07.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:06 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:07.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:06 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-09-21T12:26:07.428 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:06 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:26:08.067 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:07 smithi079 ceph-mon[30023]: Deploying daemon osd.2 on smithi079 2024-09-21T12:26:08.067 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:07 smithi079 ceph-mon[30023]: Health check cleared: CEPHADM_STRAY_DAEMON (was: 1 stray daemon(s) not managed by cephadm) 2024-09-21T12:26:08.067 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:07 smithi079 ceph-mon[30023]: Cluster is now healthy 2024-09-21T12:26:08.067 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:07 smithi079 ceph-mon[23699]: Deploying daemon osd.2 on smithi079 2024-09-21T12:26:08.067 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:07 smithi079 ceph-mon[23699]: Health check cleared: CEPHADM_STRAY_DAEMON (was: 1 stray daemon(s) not managed by cephadm) 2024-09-21T12:26:08.067 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:07 smithi079 ceph-mon[23699]: Cluster is now healthy 2024-09-21T12:26:08.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:07 smithi104 ceph-mon[25639]: Deploying daemon osd.2 on smithi079 2024-09-21T12:26:08.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:07 smithi104 ceph-mon[25639]: Health check cleared: CEPHADM_STRAY_DAEMON (was: 1 stray daemon(s) not managed by cephadm) 2024-09-21T12:26:08.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:07 smithi104 ceph-mon[25639]: Cluster is now healthy 2024-09-21T12:26:09.293 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:08 smithi079 ceph-mon[23699]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T12:26:09.293 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:08 smithi079 ceph-mon[30023]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T12:26:09.426 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:08 smithi104 ceph-mon[25639]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T12:26:10.212 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:10 smithi079 ceph-mon[23699]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T12:26:10.212 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:10 smithi079 ceph-mon[30023]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T12:26:10.426 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:10 smithi104 ceph-mon[25639]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T12:26:11.692 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:11 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:26:11.692 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:11 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:11.692 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:11 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:26:11.692 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:11 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:26:11.692 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:11 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:11.692 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:11 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:11.693 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:11 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:26:11.693 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:11 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:11.693 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:11 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:26:11.693 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:11 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:26:11.694 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:11 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:11.694 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:11 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:11.926 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:11 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:26:11.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:11 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:11.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:11 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:26:11.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:11 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:26:11.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:11 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:11.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:11 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:12.694 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:12 smithi079 ceph-mon[23699]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T12:26:12.694 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:12 smithi079 ceph-mon[30023]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T12:26:12.807 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:12 smithi104 ceph-mon[25639]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T12:26:14.063 INFO:teuthology.orchestra.run.smithi079.stdout:Created osd(s) 2 on host 'smithi079' 2024-09-21T12:26:14.176 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:14 smithi104 ceph-mon[25639]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T12:26:14.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:14 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:26:14.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:14 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:14.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:14 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-21T12:26:14.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:14 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-21T12:26:14.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:14 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:14.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:14 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-21T12:26:14.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:14 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:14.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:14 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:26:14.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:14 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:26:14.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:14 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:14.368 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:14 smithi079 ceph-mon[23699]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T12:26:14.368 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:14 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:26:14.368 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:14 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:14.368 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:14 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-21T12:26:14.368 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:14 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-21T12:26:14.368 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:14 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:14.368 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:14 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-21T12:26:14.368 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:14 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:14.368 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:14 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:26:14.368 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:14 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:26:14.368 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:14 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:14.369 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:14 smithi079 ceph-mon[30023]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T12:26:14.369 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:14 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:26:14.369 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:14 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:14.369 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:14 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-21T12:26:14.369 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:14 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-21T12:26:14.369 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:14 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:14.369 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:14 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-21T12:26:14.369 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:14 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:14.369 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:14 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:26:14.369 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:14 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:26:14.369 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:14 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:14.663 DEBUG:teuthology.orchestra.run.smithi079:osd.2> sudo journalctl -f -n 0 -u ceph-33168022-7814-11ef-baf6-efdc52797490@osd.2.service 2024-09-21T12:26:14.705 INFO:tasks.cephadm:Deploying osd.3 on smithi079 with /dev/vg_nvme/lv_1... 2024-09-21T12:26:14.705 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 33168022-7814-11ef-baf6-efdc52797490 -- lvm zap /dev/vg_nvme/lv_1 2024-09-21T12:26:14.938 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:26:15.676 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:15 smithi104 ceph-mon[25639]: Adjusting osd_memory_target on smithi079 to 5013M 2024-09-21T12:26:15.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:15 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:15.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:15 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:15.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:15 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:15.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:15 smithi104 ceph-mon[25639]: from='osd.2 [v2:172.21.15.79:6818/2982011382,v1:172.21.15.79:6819/2982011382]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-21T12:26:15.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:15 smithi079 ceph-mon[23699]: Adjusting osd_memory_target on smithi079 to 5013M 2024-09-21T12:26:15.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:15 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:15.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:15 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:15.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:15 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:15.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:15 smithi079 ceph-mon[23699]: from='osd.2 [v2:172.21.15.79:6818/2982011382,v1:172.21.15.79:6819/2982011382]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-21T12:26:15.692 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:15 smithi079 ceph-mon[30023]: Adjusting osd_memory_target on smithi079 to 5013M 2024-09-21T12:26:15.693 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:15 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:15.693 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:15 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:15.693 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:15 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:15.693 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:15 smithi079 ceph-mon[30023]: from='osd.2 [v2:172.21.15.79:6818/2982011382,v1:172.21.15.79:6819/2982011382]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-21T12:26:16.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:16 smithi104 ceph-mon[25639]: from='osd.2 [v2:172.21.15.79:6818/2982011382,v1:172.21.15.79:6819/2982011382]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-21T12:26:16.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:16 smithi104 ceph-mon[25639]: osdmap e16: 3 total, 2 up, 3 in 2024-09-21T12:26:16.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:16 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T12:26:16.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:16 smithi104 ceph-mon[25639]: from='osd.2 [v2:172.21.15.79:6818/2982011382,v1:172.21.15.79:6819/2982011382]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]: dispatch 2024-09-21T12:26:16.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:16 smithi104 ceph-mon[25639]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T12:26:16.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:16 smithi079 ceph-mon[23699]: from='osd.2 [v2:172.21.15.79:6818/2982011382,v1:172.21.15.79:6819/2982011382]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-21T12:26:16.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:16 smithi079 ceph-mon[23699]: osdmap e16: 3 total, 2 up, 3 in 2024-09-21T12:26:16.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:16 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T12:26:16.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:16 smithi079 ceph-mon[23699]: from='osd.2 [v2:172.21.15.79:6818/2982011382,v1:172.21.15.79:6819/2982011382]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]: dispatch 2024-09-21T12:26:16.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:16 smithi079 ceph-mon[23699]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T12:26:16.692 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:16 smithi079 ceph-mon[30023]: from='osd.2 [v2:172.21.15.79:6818/2982011382,v1:172.21.15.79:6819/2982011382]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-21T12:26:16.692 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:16 smithi079 ceph-mon[30023]: osdmap e16: 3 total, 2 up, 3 in 2024-09-21T12:26:16.692 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:16 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T12:26:16.693 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:16 smithi079 ceph-mon[30023]: from='osd.2 [v2:172.21.15.79:6818/2982011382,v1:172.21.15.79:6819/2982011382]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]: dispatch 2024-09-21T12:26:16.693 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:16 smithi079 ceph-mon[30023]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T12:26:17.190 INFO:journalctl@ceph.osd.2.smithi079.stdout:Sep 21 12:26:16 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-2[41222]: 2024-09-21T12:26:16.830+0000 7fb3d443e640 -1 osd.2 0 waiting for initial osdmap 2024-09-21T12:26:17.190 INFO:journalctl@ceph.osd.2.smithi079.stdout:Sep 21 12:26:16 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-2[41222]: 2024-09-21T12:26:16.834+0000 7fb3cba48640 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T12:26:17.393 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-21T12:26:17.415 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 33168022-7814-11ef-baf6-efdc52797490 -- ceph orch daemon add osd smithi079:vg_nvme/lv_1 2024-09-21T12:26:17.439 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:17 smithi079 ceph-mon[23699]: from='osd.2 [v2:172.21.15.79:6818/2982011382,v1:172.21.15.79:6819/2982011382]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]': finished 2024-09-21T12:26:17.439 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:17 smithi079 ceph-mon[23699]: osdmap e17: 3 total, 2 up, 3 in 2024-09-21T12:26:17.440 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:17 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T12:26:17.440 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:17 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T12:26:17.440 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:17 smithi079 ceph-mon[23699]: from='osd.2 [v2:172.21.15.79:6818/2982011382,v1:172.21.15.79:6819/2982011382]' entity='osd.2' 2024-09-21T12:26:17.440 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:17 smithi079 ceph-mon[30023]: from='osd.2 [v2:172.21.15.79:6818/2982011382,v1:172.21.15.79:6819/2982011382]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]': finished 2024-09-21T12:26:17.440 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:17 smithi079 ceph-mon[30023]: osdmap e17: 3 total, 2 up, 3 in 2024-09-21T12:26:17.440 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:17 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T12:26:17.440 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:17 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T12:26:17.440 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:17 smithi079 ceph-mon[30023]: from='osd.2 [v2:172.21.15.79:6818/2982011382,v1:172.21.15.79:6819/2982011382]' entity='osd.2' 2024-09-21T12:26:17.609 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:26:17.676 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:17 smithi104 ceph-mon[25639]: from='osd.2 [v2:172.21.15.79:6818/2982011382,v1:172.21.15.79:6819/2982011382]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]': finished 2024-09-21T12:26:17.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:17 smithi104 ceph-mon[25639]: osdmap e17: 3 total, 2 up, 3 in 2024-09-21T12:26:17.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:17 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T12:26:17.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:17 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T12:26:17.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:17 smithi104 ceph-mon[25639]: from='osd.2 [v2:172.21.15.79:6818/2982011382,v1:172.21.15.79:6819/2982011382]' entity='osd.2' 2024-09-21T12:26:18.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:18 smithi104 ceph-mon[25639]: purged_snaps scrub starts 2024-09-21T12:26:18.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:18 smithi104 ceph-mon[25639]: purged_snaps scrub ok 2024-09-21T12:26:18.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:18 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T12:26:18.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:18 smithi104 ceph-mon[25639]: osd.2 [v2:172.21.15.79:6818/2982011382,v1:172.21.15.79:6819/2982011382] boot 2024-09-21T12:26:18.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:18 smithi104 ceph-mon[25639]: osdmap e18: 3 total, 3 up, 3 in 2024-09-21T12:26:18.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:18 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T12:26:18.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:18 smithi104 ceph-mon[25639]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T12:26:18.692 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:18 smithi079 ceph-mon[30023]: purged_snaps scrub starts 2024-09-21T12:26:18.692 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:18 smithi079 ceph-mon[30023]: purged_snaps scrub ok 2024-09-21T12:26:18.692 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:18 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T12:26:18.692 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:18 smithi079 ceph-mon[30023]: osd.2 [v2:172.21.15.79:6818/2982011382,v1:172.21.15.79:6819/2982011382] boot 2024-09-21T12:26:18.692 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:18 smithi079 ceph-mon[30023]: osdmap e18: 3 total, 3 up, 3 in 2024-09-21T12:26:18.692 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:18 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T12:26:18.692 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:18 smithi079 ceph-mon[30023]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T12:26:18.693 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:18 smithi079 ceph-mon[23699]: purged_snaps scrub starts 2024-09-21T12:26:18.693 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:18 smithi079 ceph-mon[23699]: purged_snaps scrub ok 2024-09-21T12:26:18.693 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:18 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T12:26:18.693 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:18 smithi079 ceph-mon[23699]: osd.2 [v2:172.21.15.79:6818/2982011382,v1:172.21.15.79:6819/2982011382] boot 2024-09-21T12:26:18.693 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:18 smithi079 ceph-mon[23699]: osdmap e18: 3 total, 3 up, 3 in 2024-09-21T12:26:18.693 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:18 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T12:26:18.693 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:18 smithi079 ceph-mon[23699]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T12:26:19.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:19 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-09-21T12:26:19.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:19 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-21T12:26:19.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:19 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-21T12:26:19.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:19 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:26:19.683 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:19 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-09-21T12:26:19.683 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:19 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-21T12:26:19.683 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:19 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-21T12:26:19.683 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:19 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:26:19.683 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:19 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-09-21T12:26:19.683 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:19 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-21T12:26:19.684 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:19 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-21T12:26:19.684 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:19 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:26:20.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:20 smithi104 ceph-mon[25639]: from='client.24194 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi079:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:26:20.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:20 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-09-21T12:26:20.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:20 smithi104 ceph-mon[25639]: osdmap e19: 3 total, 3 up, 3 in 2024-09-21T12:26:20.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:20 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-09-21T12:26:20.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:20 smithi104 ceph-mon[25639]: pgmap v76: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-21T12:26:20.692 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:20 smithi079 ceph-mon[30023]: from='client.24194 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi079:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:26:20.692 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:20 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-09-21T12:26:20.692 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:20 smithi079 ceph-mon[30023]: osdmap e19: 3 total, 3 up, 3 in 2024-09-21T12:26:20.692 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:20 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-09-21T12:26:20.692 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:20 smithi079 ceph-mon[30023]: pgmap v76: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-21T12:26:20.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:20 smithi079 ceph-mon[23699]: from='client.24194 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi079:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:26:20.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:20 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-09-21T12:26:20.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:20 smithi079 ceph-mon[23699]: osdmap e19: 3 total, 3 up, 3 in 2024-09-21T12:26:20.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:20 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-09-21T12:26:20.693 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:20 smithi079 ceph-mon[23699]: pgmap v76: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-21T12:26:21.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:21 smithi104 ceph-mon[25639]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T12:26:21.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:21 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-21T12:26:21.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:21 smithi104 ceph-mon[25639]: osdmap e20: 3 total, 3 up, 3 in 2024-09-21T12:26:21.692 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:21 smithi079 ceph-mon[30023]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T12:26:21.692 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:21 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-21T12:26:21.692 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:21 smithi079 ceph-mon[30023]: osdmap e20: 3 total, 3 up, 3 in 2024-09-21T12:26:21.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:21 smithi079 ceph-mon[23699]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T12:26:21.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:21 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-21T12:26:21.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:21 smithi079 ceph-mon[23699]: osdmap e20: 3 total, 3 up, 3 in 2024-09-21T12:26:21.692 INFO:journalctl@ceph.osd.0.smithi079.stdout:Sep 21 12:26:21 smithi079 sudo[42628]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-21T12:26:21.693 INFO:journalctl@ceph.osd.0.smithi079.stdout:Sep 21 12:26:21 smithi079 sudo[42628]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T12:26:21.693 INFO:journalctl@ceph.osd.0.smithi079.stdout:Sep 21 12:26:21 smithi079 sudo[42628]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T12:26:21.693 INFO:journalctl@ceph.osd.0.smithi079.stdout:Sep 21 12:26:21 smithi079 sudo[42628]: pam_unix(sudo:session): session closed for user root 2024-09-21T12:26:21.993 INFO:journalctl@ceph.osd.0.smithi079.stdout:Sep 21 12:26:21 smithi079 sudo[42643]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-21T12:26:21.993 INFO:journalctl@ceph.osd.0.smithi079.stdout:Sep 21 12:26:21 smithi079 sudo[42643]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T12:26:21.993 INFO:journalctl@ceph.osd.0.smithi079.stdout:Sep 21 12:26:21 smithi079 sudo[42643]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T12:26:21.993 INFO:journalctl@ceph.osd.0.smithi079.stdout:Sep 21 12:26:21 smithi079 sudo[42643]: pam_unix(sudo:session): session closed for user root 2024-09-21T12:26:22.251 INFO:journalctl@ceph.osd.1.smithi079.stdout:Sep 21 12:26:21 smithi079 sudo[42687]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-21T12:26:22.251 INFO:journalctl@ceph.osd.1.smithi079.stdout:Sep 21 12:26:21 smithi079 sudo[42687]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T12:26:22.251 INFO:journalctl@ceph.osd.1.smithi079.stdout:Sep 21 12:26:21 smithi079 sudo[42687]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T12:26:22.251 INFO:journalctl@ceph.osd.1.smithi079.stdout:Sep 21 12:26:22 smithi079 sudo[42687]: pam_unix(sudo:session): session closed for user root 2024-09-21T12:26:22.515 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:22 smithi079 ceph-mon[23699]: osdmap e21: 3 total, 3 up, 3 in 2024-09-21T12:26:22.515 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:22 smithi079 ceph-mon[23699]: from='client.? 172.21.15.79:0/4272419836' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "65ac97f0-572a-400f-977b-4c20cdc0fc9f"}]: dispatch 2024-09-21T12:26:22.515 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:22 smithi079 ceph-mon[23699]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "65ac97f0-572a-400f-977b-4c20cdc0fc9f"}]: dispatch 2024-09-21T12:26:22.516 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:22 smithi079 ceph-mon[23699]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "65ac97f0-572a-400f-977b-4c20cdc0fc9f"}]': finished 2024-09-21T12:26:22.516 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:22 smithi079 ceph-mon[23699]: osdmap e22: 4 total, 3 up, 4 in 2024-09-21T12:26:22.516 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:22 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T12:26:22.516 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:22 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:22.516 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:22 smithi079 ceph-mon[23699]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-21T12:26:22.516 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:22 smithi079 ceph-mon[23699]: from='client.? 172.21.15.79:0/3016586087' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-21T12:26:22.516 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:22 smithi079 ceph-mon[30023]: osdmap e21: 3 total, 3 up, 3 in 2024-09-21T12:26:22.516 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:22 smithi079 ceph-mon[30023]: from='client.? 172.21.15.79:0/4272419836' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "65ac97f0-572a-400f-977b-4c20cdc0fc9f"}]: dispatch 2024-09-21T12:26:22.516 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:22 smithi079 ceph-mon[30023]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "65ac97f0-572a-400f-977b-4c20cdc0fc9f"}]: dispatch 2024-09-21T12:26:22.516 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:22 smithi079 ceph-mon[30023]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "65ac97f0-572a-400f-977b-4c20cdc0fc9f"}]': finished 2024-09-21T12:26:22.516 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:22 smithi079 ceph-mon[30023]: osdmap e22: 4 total, 3 up, 4 in 2024-09-21T12:26:22.516 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:22 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T12:26:22.517 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:22 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:22.517 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:22 smithi079 ceph-mon[30023]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-21T12:26:22.517 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:22 smithi079 ceph-mon[30023]: from='client.? 172.21.15.79:0/3016586087' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-21T12:26:22.517 INFO:journalctl@ceph.osd.1.smithi079.stdout:Sep 21 12:26:22 smithi079 sudo[42698]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-21T12:26:22.517 INFO:journalctl@ceph.osd.1.smithi079.stdout:Sep 21 12:26:22 smithi079 sudo[42698]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T12:26:22.517 INFO:journalctl@ceph.osd.1.smithi079.stdout:Sep 21 12:26:22 smithi079 sudo[42698]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T12:26:22.517 INFO:journalctl@ceph.osd.1.smithi079.stdout:Sep 21 12:26:22 smithi079 sudo[42698]: pam_unix(sudo:session): session closed for user root 2024-09-21T12:26:22.676 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:22 smithi104 ceph-mon[25639]: osdmap e21: 3 total, 3 up, 3 in 2024-09-21T12:26:22.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:22 smithi104 ceph-mon[25639]: from='client.? 172.21.15.79:0/4272419836' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "65ac97f0-572a-400f-977b-4c20cdc0fc9f"}]: dispatch 2024-09-21T12:26:22.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:22 smithi104 ceph-mon[25639]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "65ac97f0-572a-400f-977b-4c20cdc0fc9f"}]: dispatch 2024-09-21T12:26:22.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:22 smithi104 ceph-mon[25639]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "65ac97f0-572a-400f-977b-4c20cdc0fc9f"}]': finished 2024-09-21T12:26:22.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:22 smithi104 ceph-mon[25639]: osdmap e22: 4 total, 3 up, 4 in 2024-09-21T12:26:22.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:22 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T12:26:22.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:22 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:22.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:22 smithi104 ceph-mon[25639]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-21T12:26:22.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:22 smithi104 ceph-mon[25639]: from='client.? 172.21.15.79:0/3016586087' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-21T12:26:22.774 INFO:journalctl@ceph.osd.2.smithi079.stdout:Sep 21 12:26:22 smithi079 sudo[42742]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-21T12:26:22.774 INFO:journalctl@ceph.osd.2.smithi079.stdout:Sep 21 12:26:22 smithi079 sudo[42742]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T12:26:22.774 INFO:journalctl@ceph.osd.2.smithi079.stdout:Sep 21 12:26:22 smithi079 sudo[42742]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T12:26:22.775 INFO:journalctl@ceph.osd.2.smithi079.stdout:Sep 21 12:26:22 smithi079 sudo[42742]: pam_unix(sudo:session): session closed for user root 2024-09-21T12:26:23.031 INFO:journalctl@ceph.osd.2.smithi079.stdout:Sep 21 12:26:22 smithi079 sudo[42746]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-21T12:26:23.031 INFO:journalctl@ceph.osd.2.smithi079.stdout:Sep 21 12:26:22 smithi079 sudo[42746]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T12:26:23.031 INFO:journalctl@ceph.osd.2.smithi079.stdout:Sep 21 12:26:22 smithi079 sudo[42746]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T12:26:23.031 INFO:journalctl@ceph.osd.2.smithi079.stdout:Sep 21 12:26:22 smithi079 sudo[42746]: pam_unix(sudo:session): session closed for user root 2024-09-21T12:26:23.321 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:23 smithi079 sudo[42984]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-21T12:26:23.322 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:23 smithi079 sudo[42984]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T12:26:23.322 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:23 smithi079 sudo[42984]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T12:26:23.322 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:23 smithi079 sudo[42984]: pam_unix(sudo:session): session closed for user root 2024-09-21T12:26:23.662 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:23 smithi079 sudo[43244]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-21T12:26:23.662 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:23 smithi079 sudo[43244]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T12:26:23.662 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:23 smithi079 sudo[43244]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T12:26:23.662 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:23 smithi079 sudo[43244]: pam_unix(sudo:session): session closed for user root 2024-09-21T12:26:23.822 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:23 smithi104 sudo[29200]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-21T12:26:23.822 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:23 smithi104 sudo[29200]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T12:26:23.822 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:23 smithi104 sudo[29200]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T12:26:23.822 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:23 smithi104 sudo[29200]: pam_unix(sudo:session): session closed for user root 2024-09-21T12:26:23.918 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:23 smithi079 ceph-mon[23699]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T12:26:23.918 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:23 smithi079 ceph-mon[23699]: Cluster is now healthy 2024-09-21T12:26:23.918 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:23 smithi079 ceph-mon[23699]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-21T12:26:23.918 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:23 smithi079 ceph-mon[23699]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-21T12:26:23.919 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:23 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T12:26:23.919 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:23 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T12:26:23.919 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:23 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T12:26:23.919 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:23 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T12:26:23.919 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:23 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T12:26:23.919 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:23 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T12:26:23.919 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:23 smithi079 ceph-mon[30023]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T12:26:23.919 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:23 smithi079 ceph-mon[30023]: Cluster is now healthy 2024-09-21T12:26:23.919 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:23 smithi079 ceph-mon[30023]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-21T12:26:23.919 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:23 smithi079 ceph-mon[30023]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-21T12:26:23.919 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:23 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T12:26:23.920 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:23 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T12:26:23.920 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:23 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T12:26:23.920 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:23 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T12:26:23.920 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:23 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T12:26:23.920 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:23 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T12:26:24.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:23 smithi104 sudo[29203]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-21T12:26:24.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:23 smithi104 sudo[29203]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T12:26:24.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:23 smithi104 sudo[29203]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T12:26:24.178 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:23 smithi104 sudo[29203]: pam_unix(sudo:session): session closed for user root 2024-09-21T12:26:24.178 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:23 smithi104 ceph-mon[25639]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T12:26:24.178 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:23 smithi104 ceph-mon[25639]: Cluster is now healthy 2024-09-21T12:26:24.178 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:23 smithi104 ceph-mon[25639]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-21T12:26:24.178 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:23 smithi104 ceph-mon[25639]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-21T12:26:24.178 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:23 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T12:26:24.179 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:23 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T12:26:24.179 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:23 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T12:26:24.179 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:23 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T12:26:24.179 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:23 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T12:26:24.179 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:23 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T12:26:24.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:24 smithi079 sudo[43687]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-21T12:26:24.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:24 smithi079 sudo[43687]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T12:26:24.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:24 smithi079 sudo[43687]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T12:26:24.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:24 smithi079 sudo[43687]: pam_unix(sudo:session): session closed for user root 2024-09-21T12:26:24.692 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:24 smithi079 sudo[43776]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-21T12:26:24.692 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:24 smithi079 sudo[43776]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T12:26:24.692 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:24 smithi079 sudo[43776]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T12:26:24.692 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:24 smithi079 sudo[43776]: pam_unix(sudo:session): session closed for user root 2024-09-21T12:26:25.109 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:24 smithi079 ceph-mon[23699]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-21T12:26:25.109 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:24 smithi079 ceph-mon[23699]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-21T12:26:25.109 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:24 smithi079 ceph-mon[23699]: mgrmap e15: y(active, since 2m), standbys: x 2024-09-21T12:26:25.109 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:24 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:25.109 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:24 smithi079 ceph-mon[23699]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-21T12:26:25.109 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:24 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T12:26:25.109 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:24 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T12:26:25.109 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:24 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T12:26:25.109 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:24 smithi079 ceph-mon[23699]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-21T12:26:25.109 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:24 smithi079 ceph-mon[23699]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-21T12:26:25.109 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:24 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:25.110 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:24 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:25.110 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:24 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:25.110 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:24 smithi079 ceph-mon[30023]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-21T12:26:25.110 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:24 smithi079 ceph-mon[30023]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-21T12:26:25.110 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:24 smithi079 ceph-mon[30023]: mgrmap e15: y(active, since 2m), standbys: x 2024-09-21T12:26:25.110 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:24 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:25.110 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:24 smithi079 ceph-mon[30023]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-21T12:26:25.110 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:24 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T12:26:25.110 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:24 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T12:26:25.110 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:24 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T12:26:25.110 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:24 smithi079 ceph-mon[30023]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-21T12:26:25.111 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:24 smithi079 ceph-mon[30023]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-21T12:26:25.111 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:24 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:25.111 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:24 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:25.111 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:24 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:25.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:24 smithi104 ceph-mon[25639]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-21T12:26:25.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:24 smithi104 ceph-mon[25639]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-21T12:26:25.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:24 smithi104 ceph-mon[25639]: mgrmap e15: y(active, since 2m), standbys: x 2024-09-21T12:26:25.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:24 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:25.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:24 smithi104 ceph-mon[25639]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-21T12:26:25.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:24 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T12:26:25.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:24 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T12:26:25.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:24 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T12:26:25.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:24 smithi104 ceph-mon[25639]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-09-21T12:26:25.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:24 smithi104 ceph-mon[25639]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-21T12:26:25.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:24 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:25.178 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:24 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:25.178 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:24 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:26.426 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:26 smithi104 ceph-mon[25639]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T12:26:26.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:26 smithi079 ceph-mon[23699]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T12:26:26.442 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:26 smithi079 ceph-mon[30023]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T12:26:28.426 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:28 smithi104 ceph-mon[25639]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-09-21T12:26:28.442 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:28 smithi079 ceph-mon[30023]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-09-21T12:26:28.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:28 smithi079 ceph-mon[23699]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-09-21T12:26:29.224 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:29 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-09-21T12:26:29.224 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:29 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:26:29.224 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:29 smithi079 ceph-mon[23699]: Deploying daemon osd.3 on smithi079 2024-09-21T12:26:29.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:29 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-09-21T12:26:29.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:29 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:26:29.224 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:29 smithi079 ceph-mon[30023]: Deploying daemon osd.3 on smithi079 2024-09-21T12:26:29.426 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:29 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-09-21T12:26:29.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:29 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:26:29.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:29 smithi104 ceph-mon[25639]: Deploying daemon osd.3 on smithi079 2024-09-21T12:26:30.396 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:30 smithi079 ceph-mon[23699]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-09-21T12:26:30.397 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:30 smithi079 ceph-mon[30023]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-09-21T12:26:30.426 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:30 smithi104 ceph-mon[25639]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-09-21T12:26:32.323 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:32 smithi079 ceph-mon[23699]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-09-21T12:26:32.323 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:32 smithi079 ceph-mon[30023]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-09-21T12:26:32.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:32 smithi104 ceph-mon[25639]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-09-21T12:26:33.676 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:33 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:26:33.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:33 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:33.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:33 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:26:33.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:33 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:26:33.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:33 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:33.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:33 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:33.838 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:33 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:26:33.838 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:33 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:33.838 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:33 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:26:33.838 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:33 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:26:33.838 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:33 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:33.838 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:33 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:33.838 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:33 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:26:33.838 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:33 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:33.839 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:33 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:26:33.839 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:33 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:26:33.839 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:33 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:33.839 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:33 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:34.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:34 smithi079 ceph-mon[23699]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-09-21T12:26:34.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:34 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:34.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:34 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:34.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:34 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:34.692 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:34 smithi079 ceph-mon[30023]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-09-21T12:26:34.693 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:34 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:34.693 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:34 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:34.693 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:34 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:34.766 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:34 smithi104 ceph-mon[25639]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-09-21T12:26:34.767 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:34 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:34.767 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:34 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:34.767 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:34 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:36.100 INFO:teuthology.orchestra.run.smithi079.stdout:Created osd(s) 3 on host 'smithi079' 2024-09-21T12:26:36.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:36 smithi104 ceph-mon[25639]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-09-21T12:26:36.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:36 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:26:36.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:36 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-21T12:26:36.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:36 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:36.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:36 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-21T12:26:36.442 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:36 smithi079 ceph-mon[30023]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-09-21T12:26:36.442 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:36 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:26:36.442 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:36 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-21T12:26:36.442 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:36 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:36.442 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:36 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-21T12:26:36.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:36 smithi079 ceph-mon[23699]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-09-21T12:26:36.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:36 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:26:36.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:36 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-21T12:26:36.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:36 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:36.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:36 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-21T12:26:36.773 DEBUG:teuthology.orchestra.run.smithi079:osd.3> sudo journalctl -f -n 0 -u ceph-33168022-7814-11ef-baf6-efdc52797490@osd.3.service 2024-09-21T12:26:36.816 INFO:tasks.cephadm:Deploying osd.4 on smithi104 with /dev/vg_nvme/lv_4... 2024-09-21T12:26:36.816 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 33168022-7814-11ef-baf6-efdc52797490 -- lvm zap /dev/vg_nvme/lv_4 2024-09-21T12:26:37.013 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:26:37.420 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:37 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:37.420 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:37 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-21T12:26:37.420 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:37 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-21T12:26:37.421 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:37 smithi104 ceph-mon[25639]: Adjusting osd_memory_target on smithi079 to 3760M 2024-09-21T12:26:37.421 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:37 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:37.421 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:37 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:26:37.421 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:37 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:26:37.421 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:37 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:37.421 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:37 smithi104 ceph-mon[25639]: from='osd.3 [v2:172.21.15.79:6826/2894016442,v1:172.21.15.79:6827/2894016442]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-21T12:26:37.421 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:37 smithi104 ceph-mon[25639]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-21T12:26:37.442 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:37 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:37.442 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:37 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-21T12:26:37.442 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:37 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-21T12:26:37.442 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:37 smithi079 ceph-mon[30023]: Adjusting osd_memory_target on smithi079 to 3760M 2024-09-21T12:26:37.442 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:37 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:37.443 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:37 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:26:37.443 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:37 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:26:37.443 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:37 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:37.443 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:37 smithi079 ceph-mon[30023]: from='osd.3 [v2:172.21.15.79:6826/2894016442,v1:172.21.15.79:6827/2894016442]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-21T12:26:37.443 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:37 smithi079 ceph-mon[30023]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-21T12:26:37.443 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:37 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:37.443 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:37 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-21T12:26:37.443 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:37 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-21T12:26:37.443 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:37 smithi079 ceph-mon[23699]: Adjusting osd_memory_target on smithi079 to 3760M 2024-09-21T12:26:37.443 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:37 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:37.444 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:37 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:26:37.444 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:37 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:26:37.444 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:37 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:37.444 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:37 smithi079 ceph-mon[23699]: from='osd.3 [v2:172.21.15.79:6826/2894016442,v1:172.21.15.79:6827/2894016442]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-21T12:26:37.444 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:37 smithi079 ceph-mon[23699]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-21T12:26:38.276 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:38 smithi104 ceph-mon[25639]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-21T12:26:38.276 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:38 smithi104 ceph-mon[25639]: osdmap e23: 4 total, 3 up, 4 in 2024-09-21T12:26:38.276 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:38 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T12:26:38.276 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:38 smithi104 ceph-mon[25639]: from='osd.3 [v2:172.21.15.79:6826/2894016442,v1:172.21.15.79:6827/2894016442]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]: dispatch 2024-09-21T12:26:38.276 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:38 smithi104 ceph-mon[25639]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]: dispatch 2024-09-21T12:26:38.276 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:38 smithi104 ceph-mon[25639]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T12:26:38.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:38 smithi079 ceph-mon[23699]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-21T12:26:38.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:38 smithi079 ceph-mon[23699]: osdmap e23: 4 total, 3 up, 4 in 2024-09-21T12:26:38.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:38 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T12:26:38.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:38 smithi079 ceph-mon[23699]: from='osd.3 [v2:172.21.15.79:6826/2894016442,v1:172.21.15.79:6827/2894016442]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]: dispatch 2024-09-21T12:26:38.443 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:38 smithi079 ceph-mon[23699]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]: dispatch 2024-09-21T12:26:38.443 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:38 smithi079 ceph-mon[23699]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T12:26:38.443 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:38 smithi079 ceph-mon[30023]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-21T12:26:38.443 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:38 smithi079 ceph-mon[30023]: osdmap e23: 4 total, 3 up, 4 in 2024-09-21T12:26:38.443 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:38 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T12:26:38.443 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:38 smithi079 ceph-mon[30023]: from='osd.3 [v2:172.21.15.79:6826/2894016442,v1:172.21.15.79:6827/2894016442]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]: dispatch 2024-09-21T12:26:38.443 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:38 smithi079 ceph-mon[30023]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]: dispatch 2024-09-21T12:26:38.443 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:38 smithi079 ceph-mon[30023]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T12:26:38.443 INFO:journalctl@ceph.osd.3.smithi079.stdout:Sep 21 12:26:38 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-3[44685]: 2024-09-21T12:26:38.130+0000 7f4839766640 -1 osd.3 0 waiting for initial osdmap 2024-09-21T12:26:38.444 INFO:journalctl@ceph.osd.3.smithi079.stdout:Sep 21 12:26:38 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-3[44685]: 2024-09-21T12:26:38.135+0000 7f483558f640 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T12:26:39.341 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-21T12:26:39.359 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 33168022-7814-11ef-baf6-efdc52797490 -- ceph orch daemon add osd smithi104:vg_nvme/lv_4 2024-09-21T12:26:39.426 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:39 smithi104 ceph-mon[25639]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]': finished 2024-09-21T12:26:39.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:39 smithi104 ceph-mon[25639]: osdmap e24: 4 total, 3 up, 4 in 2024-09-21T12:26:39.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:39 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T12:26:39.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:39 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T12:26:39.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:39 smithi079 ceph-mon[23699]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]': finished 2024-09-21T12:26:39.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:39 smithi079 ceph-mon[23699]: osdmap e24: 4 total, 3 up, 4 in 2024-09-21T12:26:39.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:39 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T12:26:39.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:39 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T12:26:39.442 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:39 smithi079 ceph-mon[30023]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi079", "root=default"]}]': finished 2024-09-21T12:26:39.442 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:39 smithi079 ceph-mon[30023]: osdmap e24: 4 total, 3 up, 4 in 2024-09-21T12:26:39.442 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:39 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T12:26:39.442 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:39 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T12:26:39.551 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:26:40.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:40 smithi104 ceph-mon[25639]: purged_snaps scrub starts 2024-09-21T12:26:40.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:40 smithi104 ceph-mon[25639]: purged_snaps scrub ok 2024-09-21T12:26:40.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:40 smithi104 ceph-mon[25639]: osd.3 [v2:172.21.15.79:6826/2894016442,v1:172.21.15.79:6827/2894016442] boot 2024-09-21T12:26:40.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:40 smithi104 ceph-mon[25639]: osdmap e25: 4 total, 4 up, 4 in 2024-09-21T12:26:40.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:40 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T12:26:40.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:40 smithi104 ceph-mon[25639]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T12:26:40.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:40 smithi079 ceph-mon[23699]: purged_snaps scrub starts 2024-09-21T12:26:40.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:40 smithi079 ceph-mon[23699]: purged_snaps scrub ok 2024-09-21T12:26:40.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:40 smithi079 ceph-mon[23699]: osd.3 [v2:172.21.15.79:6826/2894016442,v1:172.21.15.79:6827/2894016442] boot 2024-09-21T12:26:40.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:40 smithi079 ceph-mon[23699]: osdmap e25: 4 total, 4 up, 4 in 2024-09-21T12:26:40.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:40 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T12:26:40.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:40 smithi079 ceph-mon[23699]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T12:26:40.442 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:40 smithi079 ceph-mon[30023]: purged_snaps scrub starts 2024-09-21T12:26:40.443 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:40 smithi079 ceph-mon[30023]: purged_snaps scrub ok 2024-09-21T12:26:40.443 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:40 smithi079 ceph-mon[30023]: osd.3 [v2:172.21.15.79:6826/2894016442,v1:172.21.15.79:6827/2894016442] boot 2024-09-21T12:26:40.443 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:40 smithi079 ceph-mon[30023]: osdmap e25: 4 total, 4 up, 4 in 2024-09-21T12:26:40.443 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:40 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T12:26:40.443 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:40 smithi079 ceph-mon[30023]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T12:26:41.426 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:41 smithi104 ceph-mon[25639]: osdmap e26: 4 total, 4 up, 4 in 2024-09-21T12:26:41.427 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:41 smithi079 ceph-mon[23699]: osdmap e26: 4 total, 4 up, 4 in 2024-09-21T12:26:41.428 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:41 smithi079 ceph-mon[30023]: osdmap e26: 4 total, 4 up, 4 in 2024-09-21T12:26:42.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:42 smithi104 ceph-mon[25639]: from='client.24208 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi104:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:26:42.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:42 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-21T12:26:42.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:42 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-21T12:26:42.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:42 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:26:42.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:42 smithi104 ceph-mon[25639]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T12:26:42.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:42 smithi079 ceph-mon[23699]: from='client.24208 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi104:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:26:42.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:42 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-21T12:26:42.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:42 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-21T12:26:42.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:42 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:26:42.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:42 smithi079 ceph-mon[23699]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T12:26:42.442 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:42 smithi079 ceph-mon[30023]: from='client.24208 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi104:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:26:42.442 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:42 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-21T12:26:42.442 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:42 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-21T12:26:42.442 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:42 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:26:42.443 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:42 smithi079 ceph-mon[30023]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T12:26:43.555 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:43 smithi079 ceph-mon[23699]: from='client.? 172.21.15.104:0/1802817660' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "278156d6-8fd6-4852-9de2-1e11ea7e8917"}]: dispatch 2024-09-21T12:26:43.555 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:43 smithi079 ceph-mon[23699]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "278156d6-8fd6-4852-9de2-1e11ea7e8917"}]: dispatch 2024-09-21T12:26:43.555 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:43 smithi079 ceph-mon[23699]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "278156d6-8fd6-4852-9de2-1e11ea7e8917"}]': finished 2024-09-21T12:26:43.555 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:43 smithi079 ceph-mon[23699]: osdmap e27: 5 total, 4 up, 5 in 2024-09-21T12:26:43.555 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:43 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T12:26:43.555 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:43 smithi079 ceph-mon[30023]: from='client.? 172.21.15.104:0/1802817660' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "278156d6-8fd6-4852-9de2-1e11ea7e8917"}]: dispatch 2024-09-21T12:26:43.555 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:43 smithi079 ceph-mon[30023]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "278156d6-8fd6-4852-9de2-1e11ea7e8917"}]: dispatch 2024-09-21T12:26:43.555 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:43 smithi079 ceph-mon[30023]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "278156d6-8fd6-4852-9de2-1e11ea7e8917"}]': finished 2024-09-21T12:26:43.555 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:43 smithi079 ceph-mon[30023]: osdmap e27: 5 total, 4 up, 5 in 2024-09-21T12:26:43.556 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:43 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T12:26:43.676 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:43 smithi104 ceph-mon[25639]: from='client.? 172.21.15.104:0/1802817660' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "278156d6-8fd6-4852-9de2-1e11ea7e8917"}]: dispatch 2024-09-21T12:26:43.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:43 smithi104 ceph-mon[25639]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "278156d6-8fd6-4852-9de2-1e11ea7e8917"}]: dispatch 2024-09-21T12:26:43.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:43 smithi104 ceph-mon[25639]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "278156d6-8fd6-4852-9de2-1e11ea7e8917"}]': finished 2024-09-21T12:26:43.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:43 smithi104 ceph-mon[25639]: osdmap e27: 5 total, 4 up, 5 in 2024-09-21T12:26:43.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:43 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T12:26:44.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:44 smithi104 ceph-mon[25639]: from='client.? 172.21.15.104:0/2616909667' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-21T12:26:44.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:44 smithi104 ceph-mon[25639]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T12:26:44.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:44 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:44.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:44 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:44.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:44 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:44.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:44 smithi079 ceph-mon[23699]: from='client.? 172.21.15.104:0/2616909667' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-21T12:26:44.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:44 smithi079 ceph-mon[23699]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T12:26:44.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:44 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:44.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:44 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:44.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:44 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:44.692 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:44 smithi079 ceph-mon[30023]: from='client.? 172.21.15.104:0/2616909667' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-21T12:26:44.693 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:44 smithi079 ceph-mon[30023]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T12:26:44.693 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:44 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:44.693 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:44 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:44.693 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:44 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:45.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:45 smithi079 ceph-mon[23699]: Detected new or changed devices on smithi079 2024-09-21T12:26:45.692 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:45 smithi079 ceph-mon[30023]: Detected new or changed devices on smithi079 2024-09-21T12:26:45.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:45 smithi104 ceph-mon[25639]: Detected new or changed devices on smithi079 2024-09-21T12:26:46.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:46 smithi079 ceph-mon[23699]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T12:26:46.692 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:46 smithi079 ceph-mon[30023]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T12:26:46.701 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:46 smithi104 ceph-mon[25639]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T12:26:48.351 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:48 smithi104 ceph-mon[25639]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T12:26:48.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:48 smithi079 ceph-mon[23699]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T12:26:48.442 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:48 smithi079 ceph-mon[30023]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T12:26:49.422 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:49 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-09-21T12:26:49.422 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:49 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:26:49.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:49 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-09-21T12:26:49.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:49 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:26:49.442 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:49 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-09-21T12:26:49.442 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:49 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:26:50.386 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:50 smithi104 ceph-mon[25639]: Deploying daemon osd.4 on smithi104 2024-09-21T12:26:50.386 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:50 smithi104 ceph-mon[25639]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T12:26:50.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:50 smithi079 ceph-mon[23699]: Deploying daemon osd.4 on smithi104 2024-09-21T12:26:50.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:50 smithi079 ceph-mon[23699]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T12:26:50.442 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:50 smithi079 ceph-mon[30023]: Deploying daemon osd.4 on smithi104 2024-09-21T12:26:50.442 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:50 smithi079 ceph-mon[30023]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T12:26:52.286 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:52 smithi104 ceph-mon[25639]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T12:26:52.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:52 smithi079 ceph-mon[23699]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T12:26:52.442 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:52 smithi079 ceph-mon[30023]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T12:26:54.300 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:54 smithi104 ceph-mon[25639]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T12:26:54.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:54 smithi079 ceph-mon[23699]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T12:26:54.442 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:54 smithi079 ceph-mon[30023]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T12:26:55.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:55 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:26:55.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:55 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:55.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:55 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:26:55.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:55 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:26:55.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:55 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:55.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:55 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:55.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:55 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:55.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:55 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:55.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:55 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:26:55.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:55 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:55.178 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:55 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:26:55.178 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:55 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:26:55.178 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:55 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:55.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:55 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:26:55.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:55 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:55.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:55 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:26:55.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:55 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:26:55.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:55 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:55.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:55 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:55.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:55 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:55.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:55 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:55.443 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:55 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:26:55.443 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:55 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:55.443 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:55 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:26:55.443 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:55 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:26:55.443 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:55 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:55.443 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:55 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:26:55.443 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:55 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:55.443 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:55 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:26:55.443 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:55 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:26:55.443 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:55 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:55.443 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:55 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:55.443 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:55 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:55.444 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:55 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:55.444 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:55 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:26:55.444 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:55 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:55.444 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:55 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:26:55.444 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:55 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:26:55.444 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:55 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:56.354 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:56 smithi104 ceph-mon[25639]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T12:26:56.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:56 smithi079 ceph-mon[23699]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T12:26:56.442 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:56 smithi079 ceph-mon[30023]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T12:26:57.176 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:57 smithi104 ceph-mon[25639]: from='osd.4 [v2:172.21.15.104:6800/3796154758,v1:172.21.15.104:6801/3796154758]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-21T12:26:57.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:57 smithi104 ceph-mon[25639]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-21T12:26:57.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:57 smithi079 ceph-mon[23699]: from='osd.4 [v2:172.21.15.104:6800/3796154758,v1:172.21.15.104:6801/3796154758]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-21T12:26:57.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:57 smithi079 ceph-mon[23699]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-21T12:26:57.442 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:57 smithi079 ceph-mon[30023]: from='osd.4 [v2:172.21.15.104:6800/3796154758,v1:172.21.15.104:6801/3796154758]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-21T12:26:57.442 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:57 smithi079 ceph-mon[30023]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-21T12:26:58.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:58 smithi104 ceph-mon[25639]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-21T12:26:58.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:58 smithi104 ceph-mon[25639]: osdmap e28: 5 total, 4 up, 5 in 2024-09-21T12:26:58.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:58 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T12:26:58.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:58 smithi104 ceph-mon[25639]: from='osd.4 [v2:172.21.15.104:6800/3796154758,v1:172.21.15.104:6801/3796154758]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-09-21T12:26:58.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:58 smithi104 ceph-mon[25639]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-09-21T12:26:58.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:58 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:58.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:58 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:58.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:58 smithi104 ceph-mon[25639]: Detected new or changed devices on smithi104 2024-09-21T12:26:58.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:58 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:26:58.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:58 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:26:58.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:58 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:58.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:58 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:26:58.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:58 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:58.178 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:58 smithi104 ceph-mon[25639]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T12:26:58.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:58 smithi079 ceph-mon[23699]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-21T12:26:58.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:58 smithi079 ceph-mon[23699]: osdmap e28: 5 total, 4 up, 5 in 2024-09-21T12:26:58.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:58 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T12:26:58.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:58 smithi079 ceph-mon[23699]: from='osd.4 [v2:172.21.15.104:6800/3796154758,v1:172.21.15.104:6801/3796154758]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-09-21T12:26:58.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:58 smithi079 ceph-mon[23699]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-09-21T12:26:58.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:58 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:58.443 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:58 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:58.443 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:58 smithi079 ceph-mon[23699]: Detected new or changed devices on smithi104 2024-09-21T12:26:58.443 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:58 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:26:58.443 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:58 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:26:58.443 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:58 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:58.443 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:58 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:26:58.443 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:58 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:58.443 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:58 smithi079 ceph-mon[23699]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T12:26:58.443 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:58 smithi079 ceph-mon[30023]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-21T12:26:58.443 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:58 smithi079 ceph-mon[30023]: osdmap e28: 5 total, 4 up, 5 in 2024-09-21T12:26:58.443 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:58 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T12:26:58.443 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:58 smithi079 ceph-mon[30023]: from='osd.4 [v2:172.21.15.104:6800/3796154758,v1:172.21.15.104:6801/3796154758]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-09-21T12:26:58.443 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:58 smithi079 ceph-mon[30023]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-09-21T12:26:58.444 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:58 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:58.444 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:58 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:58.444 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:58 smithi079 ceph-mon[30023]: Detected new or changed devices on smithi104 2024-09-21T12:26:58.444 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:58 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:26:58.444 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:58 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:26:58.444 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:58 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:58.444 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:58 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:26:58.444 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:58 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:58.444 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:58 smithi079 ceph-mon[30023]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T12:26:58.540 INFO:teuthology.orchestra.run.smithi104.stdout:Created osd(s) 4 on host 'smithi104' 2024-09-21T12:26:59.121 DEBUG:teuthology.orchestra.run.smithi104:osd.4> sudo journalctl -f -n 0 -u ceph-33168022-7814-11ef-baf6-efdc52797490@osd.4.service 2024-09-21T12:26:59.123 INFO:tasks.cephadm:Deploying osd.5 on smithi104 with /dev/vg_nvme/lv_3... 2024-09-21T12:26:59.123 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 33168022-7814-11ef-baf6-efdc52797490 -- lvm zap /dev/vg_nvme/lv_3 2024-09-21T12:26:59.356 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:26:59.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:59 smithi104 ceph-mon[25639]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]': finished 2024-09-21T12:26:59.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:59 smithi104 ceph-mon[25639]: osdmap e29: 5 total, 4 up, 5 in 2024-09-21T12:26:59.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:59 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T12:26:59.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:59 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T12:26:59.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:59 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:26:59.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:59 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:59.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:59 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-21T12:26:59.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:59 smithi104 ceph-mon[25639]: Adjusting osd_memory_target on smithi104 to 16064M 2024-09-21T12:26:59.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:59 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:59.428 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:59 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:59.428 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:59 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:26:59.428 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:59 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:26:59.428 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:59 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:59.428 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:59 smithi104 ceph-mon[25639]: from='osd.4 ' entity='osd.4' 2024-09-21T12:26:59.428 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:59 smithi104 ceph-mon[25639]: osd.4 [v2:172.21.15.104:6800/3796154758,v1:172.21.15.104:6801/3796154758] boot 2024-09-21T12:26:59.428 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:59 smithi104 ceph-mon[25639]: osdmap e30: 5 total, 5 up, 5 in 2024-09-21T12:26:59.428 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:26:59 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T12:26:59.442 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:59 smithi079 ceph-mon[30023]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]': finished 2024-09-21T12:26:59.442 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:59 smithi079 ceph-mon[30023]: osdmap e29: 5 total, 4 up, 5 in 2024-09-21T12:26:59.442 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:59 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T12:26:59.442 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:59 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T12:26:59.442 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:59 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:26:59.442 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:59 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:59.442 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:59 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-21T12:26:59.442 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:59 smithi079 ceph-mon[30023]: Adjusting osd_memory_target on smithi104 to 16064M 2024-09-21T12:26:59.443 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:59 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:59.443 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:59 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:59.443 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:59 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:26:59.443 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:59 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:26:59.443 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:59 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:59.443 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:59 smithi079 ceph-mon[30023]: from='osd.4 ' entity='osd.4' 2024-09-21T12:26:59.443 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:59 smithi079 ceph-mon[30023]: osd.4 [v2:172.21.15.104:6800/3796154758,v1:172.21.15.104:6801/3796154758] boot 2024-09-21T12:26:59.443 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:59 smithi079 ceph-mon[30023]: osdmap e30: 5 total, 5 up, 5 in 2024-09-21T12:26:59.443 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:26:59 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T12:26:59.443 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:59 smithi079 ceph-mon[23699]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]': finished 2024-09-21T12:26:59.443 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:59 smithi079 ceph-mon[23699]: osdmap e29: 5 total, 4 up, 5 in 2024-09-21T12:26:59.443 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:59 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T12:26:59.444 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:59 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T12:26:59.444 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:59 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:26:59.444 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:59 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:59.444 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:59 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-21T12:26:59.444 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:59 smithi079 ceph-mon[23699]: Adjusting osd_memory_target on smithi104 to 16064M 2024-09-21T12:26:59.444 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:59 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:59.444 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:59 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:59.444 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:59 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:26:59.444 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:59 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:26:59.444 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:59 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:26:59.444 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:59 smithi079 ceph-mon[23699]: from='osd.4 ' entity='osd.4' 2024-09-21T12:26:59.444 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:59 smithi079 ceph-mon[23699]: osd.4 [v2:172.21.15.104:6800/3796154758,v1:172.21.15.104:6801/3796154758] boot 2024-09-21T12:26:59.445 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:59 smithi079 ceph-mon[23699]: osdmap e30: 5 total, 5 up, 5 in 2024-09-21T12:26:59.445 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:26:59 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T12:27:00.401 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:00 smithi104 ceph-mon[25639]: purged_snaps scrub starts 2024-09-21T12:27:00.402 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:00 smithi104 ceph-mon[25639]: purged_snaps scrub ok 2024-09-21T12:27:00.402 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:00 smithi104 ceph-mon[25639]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-21T12:27:00.402 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:00 smithi104 ceph-mon[25639]: osdmap e31: 5 total, 5 up, 5 in 2024-09-21T12:27:00.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:00 smithi079 ceph-mon[23699]: purged_snaps scrub starts 2024-09-21T12:27:00.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:00 smithi079 ceph-mon[23699]: purged_snaps scrub ok 2024-09-21T12:27:00.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:00 smithi079 ceph-mon[23699]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-21T12:27:00.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:00 smithi079 ceph-mon[23699]: osdmap e31: 5 total, 5 up, 5 in 2024-09-21T12:27:00.442 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:00 smithi079 ceph-mon[30023]: purged_snaps scrub starts 2024-09-21T12:27:00.442 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:00 smithi079 ceph-mon[30023]: purged_snaps scrub ok 2024-09-21T12:27:00.443 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:00 smithi079 ceph-mon[30023]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-21T12:27:00.443 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:00 smithi079 ceph-mon[30023]: osdmap e31: 5 total, 5 up, 5 in 2024-09-21T12:27:01.779 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-21T12:27:01.797 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 33168022-7814-11ef-baf6-efdc52797490 -- ceph orch daemon add osd smithi104:vg_nvme/lv_3 2024-09-21T12:27:01.987 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:27:02.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:02 smithi079 ceph-mon[23699]: osdmap e32: 5 total, 5 up, 5 in 2024-09-21T12:27:02.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:02 smithi079 ceph-mon[23699]: osdmap e33: 5 total, 5 up, 5 in 2024-09-21T12:27:02.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:02 smithi079 ceph-mon[23699]: pgmap v111: 1 pgs: 1 peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 148 KiB/s, 0 objects/s recovering 2024-09-21T12:27:02.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:02 smithi079 ceph-mon[30023]: osdmap e32: 5 total, 5 up, 5 in 2024-09-21T12:27:02.193 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:02 smithi079 ceph-mon[30023]: osdmap e33: 5 total, 5 up, 5 in 2024-09-21T12:27:02.193 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:02 smithi079 ceph-mon[30023]: pgmap v111: 1 pgs: 1 peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 148 KiB/s, 0 objects/s recovering 2024-09-21T12:27:02.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:02 smithi104 ceph-mon[25639]: osdmap e32: 5 total, 5 up, 5 in 2024-09-21T12:27:02.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:02 smithi104 ceph-mon[25639]: osdmap e33: 5 total, 5 up, 5 in 2024-09-21T12:27:02.411 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:02 smithi104 ceph-mon[25639]: pgmap v111: 1 pgs: 1 peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 148 KiB/s, 0 objects/s recovering 2024-09-21T12:27:03.926 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:03 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:27:03.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:03 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:27:03.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:03 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:27:03.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:03 smithi104 ceph-mon[25639]: osdmap e34: 5 total, 5 up, 5 in 2024-09-21T12:27:03.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:03 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-21T12:27:03.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:03 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-21T12:27:03.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:03 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:27:03.942 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:03 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:27:03.942 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:03 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:27:03.942 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:03 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:27:03.942 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:03 smithi079 ceph-mon[30023]: osdmap e34: 5 total, 5 up, 5 in 2024-09-21T12:27:03.942 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:03 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-21T12:27:03.942 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:03 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-21T12:27:03.942 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:03 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:27:03.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:03 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:27:03.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:03 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:27:03.943 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:03 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:27:03.943 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:03 smithi079 ceph-mon[23699]: osdmap e34: 5 total, 5 up, 5 in 2024-09-21T12:27:03.943 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:03 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-21T12:27:03.943 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:03 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-21T12:27:03.943 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:03 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:27:04.926 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:04 smithi104 ceph-mon[25639]: from='client.24235 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi104:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:27:04.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:04 smithi104 ceph-mon[25639]: pgmap v113: 1 pgs: 1 peering; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-09-21T12:27:04.930 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:04 smithi079 ceph-mon[23699]: from='client.24235 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi104:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:27:04.930 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:04 smithi079 ceph-mon[23699]: pgmap v113: 1 pgs: 1 peering; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-09-21T12:27:04.930 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:04 smithi079 ceph-mon[30023]: from='client.24235 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi104:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:27:04.930 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:04 smithi079 ceph-mon[30023]: pgmap v113: 1 pgs: 1 peering; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-09-21T12:27:06.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:05 smithi104 ceph-mon[25639]: from='client.? 172.21.15.104:0/339764014' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5435eabf-f4d1-4c86-b334-fa3acc9162c4"}]: dispatch 2024-09-21T12:27:06.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:05 smithi104 ceph-mon[25639]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5435eabf-f4d1-4c86-b334-fa3acc9162c4"}]: dispatch 2024-09-21T12:27:06.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:05 smithi104 ceph-mon[25639]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5435eabf-f4d1-4c86-b334-fa3acc9162c4"}]': finished 2024-09-21T12:27:06.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:05 smithi104 ceph-mon[25639]: osdmap e35: 6 total, 5 up, 6 in 2024-09-21T12:27:06.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:05 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T12:27:06.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:05 smithi079 ceph-mon[23699]: from='client.? 172.21.15.104:0/339764014' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5435eabf-f4d1-4c86-b334-fa3acc9162c4"}]: dispatch 2024-09-21T12:27:06.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:05 smithi079 ceph-mon[23699]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5435eabf-f4d1-4c86-b334-fa3acc9162c4"}]: dispatch 2024-09-21T12:27:06.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:05 smithi079 ceph-mon[23699]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5435eabf-f4d1-4c86-b334-fa3acc9162c4"}]': finished 2024-09-21T12:27:06.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:05 smithi079 ceph-mon[23699]: osdmap e35: 6 total, 5 up, 6 in 2024-09-21T12:27:06.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:05 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T12:27:06.193 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:05 smithi079 ceph-mon[30023]: from='client.? 172.21.15.104:0/339764014' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5435eabf-f4d1-4c86-b334-fa3acc9162c4"}]: dispatch 2024-09-21T12:27:06.193 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:05 smithi079 ceph-mon[30023]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5435eabf-f4d1-4c86-b334-fa3acc9162c4"}]: dispatch 2024-09-21T12:27:06.193 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:05 smithi079 ceph-mon[30023]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5435eabf-f4d1-4c86-b334-fa3acc9162c4"}]': finished 2024-09-21T12:27:06.193 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:05 smithi079 ceph-mon[30023]: osdmap e35: 6 total, 5 up, 6 in 2024-09-21T12:27:06.193 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:05 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T12:27:07.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:06 smithi104 ceph-mon[25639]: pgmap v115: 1 pgs: 1 peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 118 KiB/s, 0 objects/s recovering 2024-09-21T12:27:07.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:06 smithi104 ceph-mon[25639]: from='client.? 172.21.15.104:0/755898449' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-21T12:27:07.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:06 smithi079 ceph-mon[23699]: pgmap v115: 1 pgs: 1 peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 118 KiB/s, 0 objects/s recovering 2024-09-21T12:27:07.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:06 smithi079 ceph-mon[23699]: from='client.? 172.21.15.104:0/755898449' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-21T12:27:07.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:06 smithi079 ceph-mon[30023]: pgmap v115: 1 pgs: 1 peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 118 KiB/s, 0 objects/s recovering 2024-09-21T12:27:07.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:06 smithi079 ceph-mon[30023]: from='client.? 172.21.15.104:0/755898449' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-21T12:27:08.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:08 smithi104 ceph-mon[25639]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 92 KiB/s, 0 objects/s recovering 2024-09-21T12:27:08.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:08 smithi079 ceph-mon[23699]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 92 KiB/s, 0 objects/s recovering 2024-09-21T12:27:08.442 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:08 smithi079 ceph-mon[30023]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 92 KiB/s, 0 objects/s recovering 2024-09-21T12:27:10.287 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:10 smithi104 ceph-mon[25639]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-21T12:27:10.360 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:10 smithi079 ceph-mon[23699]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-21T12:27:10.360 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:10 smithi079 ceph-mon[30023]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-21T12:27:12.086 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:11 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-09-21T12:27:12.087 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:11 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:27:12.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:11 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-09-21T12:27:12.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:11 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:27:12.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:11 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-09-21T12:27:12.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:11 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:27:13.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:12 smithi104 ceph-mon[25639]: Deploying daemon osd.5 on smithi104 2024-09-21T12:27:13.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:12 smithi104 ceph-mon[25639]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-21T12:27:13.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:12 smithi079 ceph-mon[23699]: Deploying daemon osd.5 on smithi104 2024-09-21T12:27:13.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:12 smithi079 ceph-mon[23699]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-21T12:27:13.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:12 smithi079 ceph-mon[30023]: Deploying daemon osd.5 on smithi104 2024-09-21T12:27:13.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:12 smithi079 ceph-mon[30023]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-21T12:27:14.399 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:14 smithi104 ceph-mon[25639]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-21T12:27:14.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:14 smithi079 ceph-mon[23699]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-21T12:27:14.442 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:14 smithi079 ceph-mon[30023]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-21T12:27:16.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:16 smithi104 ceph-mon[25639]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-21T12:27:16.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:16 smithi079 ceph-mon[23699]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-21T12:27:16.442 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:16 smithi079 ceph-mon[30023]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-21T12:27:17.817 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:17 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:27:17.817 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:17 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:27:17.817 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:17 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:27:17.817 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:17 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:27:17.817 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:17 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:27:17.817 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:17 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:27:17.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:17 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:27:17.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:17 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:27:17.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:17 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:27:17.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:17 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:27:17.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:17 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:27:17.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:17 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:27:17.942 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:17 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:27:17.943 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:17 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:27:17.943 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:17 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:27:17.943 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:17 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:27:17.943 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:17 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:27:17.943 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:17 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:27:19.234 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:19 smithi104 ceph-mon[25639]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-21T12:27:19.234 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:19 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:27:19.234 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:19 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:27:19.234 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:19 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:27:19.234 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:19 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:27:19.235 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:19 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:27:19.235 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:19 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:27:19.235 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:19 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:27:19.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:19 smithi079 ceph-mon[23699]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-21T12:27:19.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:19 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:27:19.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:19 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:27:19.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:19 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:27:19.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:19 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:27:19.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:19 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:27:19.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:19 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:27:19.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:19 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:27:19.443 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:19 smithi079 ceph-mon[30023]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-21T12:27:19.443 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:19 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:27:19.443 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:19 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:27:19.443 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:19 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:27:19.443 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:19 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:27:19.443 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:19 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:27:19.443 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:19 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:27:19.443 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:19 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:27:21.058 INFO:teuthology.orchestra.run.smithi104.stdout:Created osd(s) 5 on host 'smithi104' 2024-09-21T12:27:21.287 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:21 smithi104 ceph-mon[25639]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-21T12:27:21.287 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:21 smithi104 ceph-mon[25639]: from='osd.5 [v2:172.21.15.104:6808/189336043,v1:172.21.15.104:6809/189336043]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-21T12:27:21.287 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:21 smithi104 ceph-mon[25639]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-21T12:27:21.326 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:21 smithi079 ceph-mon[23699]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-21T12:27:21.326 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:21 smithi079 ceph-mon[23699]: from='osd.5 [v2:172.21.15.104:6808/189336043,v1:172.21.15.104:6809/189336043]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-21T12:27:21.326 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:21 smithi079 ceph-mon[23699]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-21T12:27:21.326 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:21 smithi079 ceph-mon[30023]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-21T12:27:21.326 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:21 smithi079 ceph-mon[30023]: from='osd.5 [v2:172.21.15.104:6808/189336043,v1:172.21.15.104:6809/189336043]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-21T12:27:21.326 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:21 smithi079 ceph-mon[30023]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-21T12:27:22.040 DEBUG:teuthology.orchestra.run.smithi104:osd.5> sudo journalctl -f -n 0 -u ceph-33168022-7814-11ef-baf6-efdc52797490@osd.5.service 2024-09-21T12:27:22.043 INFO:tasks.cephadm:Deploying osd.6 on smithi104 with /dev/vg_nvme/lv_2... 2024-09-21T12:27:22.043 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 33168022-7814-11ef-baf6-efdc52797490 -- lvm zap /dev/vg_nvme/lv_2 2024-09-21T12:27:22.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:22 smithi079 ceph-mon[23699]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-21T12:27:22.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:22 smithi079 ceph-mon[23699]: osdmap e36: 6 total, 5 up, 6 in 2024-09-21T12:27:22.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:22 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T12:27:22.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:22 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:27:22.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:22 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:27:22.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:22 smithi079 ceph-mon[23699]: from='osd.5 [v2:172.21.15.104:6808/189336043,v1:172.21.15.104:6809/189336043]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-09-21T12:27:22.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:22 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:27:22.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:22 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:27:22.193 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:22 smithi079 ceph-mon[23699]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-09-21T12:27:22.193 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:22 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:27:22.193 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:22 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:27:22.193 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:22 smithi079 ceph-mon[30023]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-21T12:27:22.193 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:22 smithi079 ceph-mon[30023]: osdmap e36: 6 total, 5 up, 6 in 2024-09-21T12:27:22.193 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:22 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T12:27:22.193 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:22 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:27:22.193 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:22 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:27:22.193 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:22 smithi079 ceph-mon[30023]: from='osd.5 [v2:172.21.15.104:6808/189336043,v1:172.21.15.104:6809/189336043]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-09-21T12:27:22.193 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:22 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:27:22.193 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:22 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:27:22.193 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:22 smithi079 ceph-mon[30023]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-09-21T12:27:22.194 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:22 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:27:22.194 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:22 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:27:22.277 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:22 smithi104 ceph-mon[25639]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-21T12:27:22.277 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:22 smithi104 ceph-mon[25639]: osdmap e36: 6 total, 5 up, 6 in 2024-09-21T12:27:22.277 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:22 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T12:27:22.277 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:22 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:27:22.277 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:22 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:27:22.278 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:22 smithi104 ceph-mon[25639]: from='osd.5 [v2:172.21.15.104:6808/189336043,v1:172.21.15.104:6809/189336043]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-09-21T12:27:22.278 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:22 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:27:22.278 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:22 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:27:22.278 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:22 smithi104 ceph-mon[25639]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-09-21T12:27:22.278 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:22 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:27:22.278 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:22 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:27:22.302 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:27:22.531 INFO:journalctl@ceph.osd.5.smithi104.stdout:Sep 21 12:27:22 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-5[35230]: 2024-09-21T12:27:22.482+0000 7f81586b2640 -1 osd.5 0 waiting for initial osdmap 2024-09-21T12:27:22.531 INFO:journalctl@ceph.osd.5.smithi104.stdout:Sep 21 12:27:22 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-5[35230]: 2024-09-21T12:27:22.489+0000 7f81504d3640 -1 osd.5 37 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T12:27:23.368 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:23 smithi104 ceph-mon[25639]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-21T12:27:23.368 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:23 smithi104 ceph-mon[25639]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]': finished 2024-09-21T12:27:23.368 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:23 smithi104 ceph-mon[25639]: osdmap e37: 6 total, 5 up, 6 in 2024-09-21T12:27:23.369 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:23 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T12:27:23.369 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:23 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T12:27:23.369 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:23 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:27:23.369 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:23 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:27:23.369 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:23 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:27:23.369 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:23 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:27:23.369 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:23 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:27:23.369 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:23 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:27:23.369 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:23 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:27:23.369 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:23 smithi104 ceph-mon[25639]: from='osd.5 ' entity='osd.5' 2024-09-21T12:27:23.369 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:23 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:27:23.369 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:23 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:27:23.370 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:23 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:27:23.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:23 smithi079 ceph-mon[23699]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-21T12:27:23.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:23 smithi079 ceph-mon[23699]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]': finished 2024-09-21T12:27:23.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:23 smithi079 ceph-mon[23699]: osdmap e37: 6 total, 5 up, 6 in 2024-09-21T12:27:23.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:23 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T12:27:23.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:23 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T12:27:23.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:23 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:27:23.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:23 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:27:23.443 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:23 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:27:23.443 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:23 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:27:23.443 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:23 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:27:23.443 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:23 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:27:23.443 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:23 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:27:23.443 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:23 smithi079 ceph-mon[23699]: from='osd.5 ' entity='osd.5' 2024-09-21T12:27:23.443 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:23 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:27:23.443 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:23 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:27:23.443 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:23 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:27:23.443 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:23 smithi079 ceph-mon[30023]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-21T12:27:23.443 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:23 smithi079 ceph-mon[30023]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]': finished 2024-09-21T12:27:23.444 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:23 smithi079 ceph-mon[30023]: osdmap e37: 6 total, 5 up, 6 in 2024-09-21T12:27:23.444 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:23 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T12:27:23.444 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:23 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T12:27:23.444 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:23 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:27:23.444 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:23 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:27:23.444 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:23 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:27:23.444 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:23 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:27:23.444 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:23 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:27:23.444 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:23 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:27:23.444 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:23 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:27:23.444 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:23 smithi079 ceph-mon[30023]: from='osd.5 ' entity='osd.5' 2024-09-21T12:27:23.444 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:23 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:27:23.444 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:23 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:27:23.444 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:23 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:27:24.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:24 smithi104 ceph-mon[25639]: purged_snaps scrub starts 2024-09-21T12:27:24.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:24 smithi104 ceph-mon[25639]: purged_snaps scrub ok 2024-09-21T12:27:24.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:24 smithi104 ceph-mon[25639]: Detected new or changed devices on smithi104 2024-09-21T12:27:24.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:24 smithi104 ceph-mon[25639]: osd.5 [v2:172.21.15.104:6808/189336043,v1:172.21.15.104:6809/189336043] boot 2024-09-21T12:27:24.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:24 smithi104 ceph-mon[25639]: osdmap e38: 6 total, 6 up, 6 in 2024-09-21T12:27:24.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:24 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T12:27:24.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:24 smithi104 ceph-mon[25639]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T12:27:24.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:24 smithi079 ceph-mon[23699]: purged_snaps scrub starts 2024-09-21T12:27:24.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:24 smithi079 ceph-mon[23699]: purged_snaps scrub ok 2024-09-21T12:27:24.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:24 smithi079 ceph-mon[23699]: Detected new or changed devices on smithi104 2024-09-21T12:27:24.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:24 smithi079 ceph-mon[23699]: osd.5 [v2:172.21.15.104:6808/189336043,v1:172.21.15.104:6809/189336043] boot 2024-09-21T12:27:24.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:24 smithi079 ceph-mon[23699]: osdmap e38: 6 total, 6 up, 6 in 2024-09-21T12:27:24.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:24 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T12:27:24.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:24 smithi079 ceph-mon[23699]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T12:27:24.442 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:24 smithi079 ceph-mon[30023]: purged_snaps scrub starts 2024-09-21T12:27:24.442 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:24 smithi079 ceph-mon[30023]: purged_snaps scrub ok 2024-09-21T12:27:24.442 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:24 smithi079 ceph-mon[30023]: Detected new or changed devices on smithi104 2024-09-21T12:27:24.443 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:24 smithi079 ceph-mon[30023]: osd.5 [v2:172.21.15.104:6808/189336043,v1:172.21.15.104:6809/189336043] boot 2024-09-21T12:27:24.443 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:24 smithi079 ceph-mon[30023]: osdmap e38: 6 total, 6 up, 6 in 2024-09-21T12:27:24.443 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:24 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T12:27:24.443 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:24 smithi079 ceph-mon[30023]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T12:27:24.778 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-21T12:27:24.798 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 33168022-7814-11ef-baf6-efdc52797490 -- ceph orch daemon add osd smithi104:vg_nvme/lv_2 2024-09-21T12:27:24.992 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:27:25.390 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:25 smithi104 ceph-mon[25639]: osdmap e39: 6 total, 6 up, 6 in 2024-09-21T12:27:25.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:25 smithi079 ceph-mon[23699]: osdmap e39: 6 total, 6 up, 6 in 2024-09-21T12:27:25.442 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:25 smithi079 ceph-mon[30023]: osdmap e39: 6 total, 6 up, 6 in 2024-09-21T12:27:26.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:26 smithi104 ceph-mon[25639]: osdmap e40: 6 total, 6 up, 6 in 2024-09-21T12:27:26.320 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:26 smithi104 ceph-mon[25639]: pgmap v130: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T12:27:26.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:26 smithi079 ceph-mon[23699]: osdmap e40: 6 total, 6 up, 6 in 2024-09-21T12:27:26.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:26 smithi079 ceph-mon[23699]: pgmap v130: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T12:27:26.442 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:26 smithi079 ceph-mon[30023]: osdmap e40: 6 total, 6 up, 6 in 2024-09-21T12:27:26.442 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:26 smithi079 ceph-mon[30023]: pgmap v130: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T12:27:27.344 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:27 smithi104 ceph-mon[25639]: osdmap e41: 6 total, 6 up, 6 in 2024-09-21T12:27:27.345 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:27 smithi104 ceph-mon[25639]: from='client.24262 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi104:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:27:27.345 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:27 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-21T12:27:27.345 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:27 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-21T12:27:27.345 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:27 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:27:27.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:27 smithi079 ceph-mon[23699]: osdmap e41: 6 total, 6 up, 6 in 2024-09-21T12:27:27.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:27 smithi079 ceph-mon[23699]: from='client.24262 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi104:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:27:27.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:27 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-21T12:27:27.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:27 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-21T12:27:27.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:27 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:27:27.442 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:27 smithi079 ceph-mon[30023]: osdmap e41: 6 total, 6 up, 6 in 2024-09-21T12:27:27.442 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:27 smithi079 ceph-mon[30023]: from='client.24262 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi104:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:27:27.442 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:27 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-21T12:27:27.443 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:27 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-21T12:27:27.443 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:27 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:27:28.208 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:28 smithi104 ceph-mon[25639]: osdmap e42: 6 total, 6 up, 6 in 2024-09-21T12:27:28.208 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:28 smithi104 ceph-mon[25639]: pgmap v133: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T12:27:28.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:28 smithi079 ceph-mon[23699]: osdmap e42: 6 total, 6 up, 6 in 2024-09-21T12:27:28.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:28 smithi079 ceph-mon[23699]: pgmap v133: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T12:27:28.442 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:28 smithi079 ceph-mon[30023]: osdmap e42: 6 total, 6 up, 6 in 2024-09-21T12:27:28.442 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:28 smithi079 ceph-mon[30023]: pgmap v133: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T12:27:29.426 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:29 smithi104 ceph-mon[25639]: from='client.? 172.21.15.104:0/3135899325' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "df36e40d-a08e-4b0f-8774-4862209d0ba0"}]: dispatch 2024-09-21T12:27:29.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:29 smithi104 ceph-mon[25639]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "df36e40d-a08e-4b0f-8774-4862209d0ba0"}]: dispatch 2024-09-21T12:27:29.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:29 smithi104 ceph-mon[25639]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "df36e40d-a08e-4b0f-8774-4862209d0ba0"}]': finished 2024-09-21T12:27:29.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:29 smithi104 ceph-mon[25639]: osdmap e43: 7 total, 6 up, 7 in 2024-09-21T12:27:29.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:29 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T12:27:29.441 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:29 smithi079 ceph-mon[23699]: from='client.? 172.21.15.104:0/3135899325' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "df36e40d-a08e-4b0f-8774-4862209d0ba0"}]: dispatch 2024-09-21T12:27:29.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:29 smithi079 ceph-mon[23699]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "df36e40d-a08e-4b0f-8774-4862209d0ba0"}]: dispatch 2024-09-21T12:27:29.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:29 smithi079 ceph-mon[23699]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "df36e40d-a08e-4b0f-8774-4862209d0ba0"}]': finished 2024-09-21T12:27:29.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:29 smithi079 ceph-mon[23699]: osdmap e43: 7 total, 6 up, 7 in 2024-09-21T12:27:29.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:29 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T12:27:29.442 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:29 smithi079 ceph-mon[30023]: from='client.? 172.21.15.104:0/3135899325' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "df36e40d-a08e-4b0f-8774-4862209d0ba0"}]: dispatch 2024-09-21T12:27:29.442 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:29 smithi079 ceph-mon[30023]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "df36e40d-a08e-4b0f-8774-4862209d0ba0"}]: dispatch 2024-09-21T12:27:29.442 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:29 smithi079 ceph-mon[30023]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "df36e40d-a08e-4b0f-8774-4862209d0ba0"}]': finished 2024-09-21T12:27:29.442 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:29 smithi079 ceph-mon[30023]: osdmap e43: 7 total, 6 up, 7 in 2024-09-21T12:27:29.442 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:29 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T12:27:30.426 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:30 smithi104 ceph-mon[25639]: from='client.? 172.21.15.104:0/3752087652' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-21T12:27:30.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:30 smithi104 ceph-mon[25639]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T12:27:30.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:30 smithi079 ceph-mon[23699]: from='client.? 172.21.15.104:0/3752087652' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-21T12:27:30.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:30 smithi079 ceph-mon[23699]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T12:27:30.442 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:30 smithi079 ceph-mon[30023]: from='client.? 172.21.15.104:0/3752087652' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-21T12:27:30.442 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:30 smithi079 ceph-mon[30023]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T12:27:32.273 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:32 smithi104 ceph-mon[25639]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T12:27:32.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:32 smithi079 ceph-mon[23699]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T12:27:32.442 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:32 smithi079 ceph-mon[30023]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T12:27:34.386 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:34 smithi104 ceph-mon[25639]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T12:27:34.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:34 smithi079 ceph-mon[23699]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T12:27:34.442 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:34 smithi079 ceph-mon[30023]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T12:27:35.389 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:35 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-09-21T12:27:35.389 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:35 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:27:35.389 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:35 smithi104 ceph-mon[25639]: Deploying daemon osd.6 on smithi104 2024-09-21T12:27:35.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:35 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-09-21T12:27:35.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:35 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:27:35.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:35 smithi079 ceph-mon[23699]: Deploying daemon osd.6 on smithi104 2024-09-21T12:27:35.442 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:35 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-09-21T12:27:35.442 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:35 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:27:35.442 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:35 smithi079 ceph-mon[30023]: Deploying daemon osd.6 on smithi104 2024-09-21T12:27:36.225 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:36 smithi104 ceph-mon[25639]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T12:27:36.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:36 smithi079 ceph-mon[23699]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T12:27:36.442 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:36 smithi079 ceph-mon[30023]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T12:27:38.215 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:38 smithi104 ceph-mon[25639]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T12:27:38.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:38 smithi079 ceph-mon[23699]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T12:27:38.442 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:38 smithi079 ceph-mon[30023]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T12:27:40.155 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:40 smithi104 ceph-mon[25639]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T12:27:40.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:40 smithi079 ceph-mon[23699]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T12:27:40.442 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:40 smithi079 ceph-mon[30023]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T12:27:41.370 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:41 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:27:41.370 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:41 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:27:41.370 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:41 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:27:41.370 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:41 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:27:41.370 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:41 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:27:41.371 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:41 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:27:41.371 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:41 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:27:41.371 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:41 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:27:41.371 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:41 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:27:41.371 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:41 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:27:41.371 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:41 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:27:41.371 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:41 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:27:41.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:41 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:27:41.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:41 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:27:41.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:41 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:27:41.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:41 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:27:41.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:41 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:27:41.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:41 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:27:42.397 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:42 smithi079 ceph-mon[23699]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T12:27:42.397 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:42 smithi079 ceph-mon[30023]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T12:27:42.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:42 smithi104 ceph-mon[25639]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T12:27:43.400 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:43 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:27:43.400 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:43 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:27:43.400 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:43 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:27:43.401 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:43 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:27:43.401 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:43 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:27:43.401 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:43 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:27:43.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:43 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:27:43.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:43 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:27:43.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:43 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:27:43.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:43 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:27:43.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:43 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:27:43.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:43 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:27:43.692 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:43 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:27:43.692 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:43 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:27:43.693 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:43 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:27:43.693 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:43 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:27:43.693 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:43 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:27:43.693 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:43 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:27:44.412 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:44 smithi104 ceph-mon[25639]: from='osd.6 [v2:172.21.15.104:6816/2903175260,v1:172.21.15.104:6817/2903175260]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-21T12:27:44.412 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:44 smithi104 ceph-mon[25639]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-21T12:27:44.413 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:44 smithi104 ceph-mon[25639]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T12:27:44.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:44 smithi079 ceph-mon[23699]: from='osd.6 [v2:172.21.15.104:6816/2903175260,v1:172.21.15.104:6817/2903175260]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-21T12:27:44.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:44 smithi079 ceph-mon[23699]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-21T12:27:44.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:44 smithi079 ceph-mon[23699]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T12:27:44.692 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:44 smithi079 ceph-mon[30023]: from='osd.6 [v2:172.21.15.104:6816/2903175260,v1:172.21.15.104:6817/2903175260]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-21T12:27:44.692 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:44 smithi079 ceph-mon[30023]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-21T12:27:44.692 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:44 smithi079 ceph-mon[30023]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T12:27:45.039 INFO:teuthology.orchestra.run.smithi104.stdout:Created osd(s) 6 on host 'smithi104' 2024-09-21T12:27:45.353 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:45 smithi104 ceph-mon[25639]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-21T12:27:45.353 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:45 smithi104 ceph-mon[25639]: osdmap e44: 7 total, 6 up, 7 in 2024-09-21T12:27:45.353 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:45 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T12:27:45.354 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:45 smithi104 ceph-mon[25639]: from='osd.6 [v2:172.21.15.104:6816/2903175260,v1:172.21.15.104:6817/2903175260]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-09-21T12:27:45.354 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:45 smithi104 ceph-mon[25639]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-09-21T12:27:45.354 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:45 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:27:45.354 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:45 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:27:45.354 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:45 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-21T12:27:45.354 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:45 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-21T12:27:45.354 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:45 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:27:45.354 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:45 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-21T12:27:45.354 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:45 smithi104 ceph-mon[25639]: Adjusting osd_memory_target on smithi104 to 5354M 2024-09-21T12:27:45.354 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:45 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:27:45.354 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:45 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:27:45.354 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:45 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:27:45.354 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:45 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:27:45.652 DEBUG:teuthology.orchestra.run.smithi104:osd.6> sudo journalctl -f -n 0 -u ceph-33168022-7814-11ef-baf6-efdc52797490@osd.6.service 2024-09-21T12:27:45.654 INFO:tasks.cephadm:Deploying osd.7 on smithi104 with /dev/vg_nvme/lv_1... 2024-09-21T12:27:45.654 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 33168022-7814-11ef-baf6-efdc52797490 -- lvm zap /dev/vg_nvme/lv_1 2024-09-21T12:27:45.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:45 smithi079 ceph-mon[23699]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-21T12:27:45.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:45 smithi079 ceph-mon[23699]: osdmap e44: 7 total, 6 up, 7 in 2024-09-21T12:27:45.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:45 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T12:27:45.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:45 smithi079 ceph-mon[23699]: from='osd.6 [v2:172.21.15.104:6816/2903175260,v1:172.21.15.104:6817/2903175260]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-09-21T12:27:45.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:45 smithi079 ceph-mon[23699]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-09-21T12:27:45.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:45 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:27:45.693 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:45 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:27:45.693 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:45 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-21T12:27:45.693 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:45 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-21T12:27:45.693 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:45 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:27:45.693 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:45 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-21T12:27:45.693 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:45 smithi079 ceph-mon[23699]: Adjusting osd_memory_target on smithi104 to 5354M 2024-09-21T12:27:45.693 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:45 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:27:45.693 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:45 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:27:45.693 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:45 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:27:45.693 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:45 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:27:45.693 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:45 smithi079 ceph-mon[30023]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-21T12:27:45.693 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:45 smithi079 ceph-mon[30023]: osdmap e44: 7 total, 6 up, 7 in 2024-09-21T12:27:45.694 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:45 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T12:27:45.694 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:45 smithi079 ceph-mon[30023]: from='osd.6 [v2:172.21.15.104:6816/2903175260,v1:172.21.15.104:6817/2903175260]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-09-21T12:27:45.694 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:45 smithi079 ceph-mon[30023]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-09-21T12:27:45.694 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:45 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:27:45.694 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:45 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:27:45.694 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:45 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-21T12:27:45.694 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:45 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-21T12:27:45.694 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:45 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:27:45.694 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:45 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-21T12:27:45.694 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:45 smithi079 ceph-mon[30023]: Adjusting osd_memory_target on smithi104 to 5354M 2024-09-21T12:27:45.694 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:45 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:27:45.694 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:45 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:27:45.694 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:45 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:27:45.695 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:45 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:27:45.885 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:27:45.927 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 12:27:45 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-6[38758]: 2024-09-21T12:27:45.680+0000 7f1987fcc640 -1 osd.6 0 waiting for initial osdmap 2024-09-21T12:27:45.927 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 12:27:45 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-6[38758]: 2024-09-21T12:27:45.687+0000 7f197fded640 -1 osd.6 45 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T12:27:46.316 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:46 smithi104 ceph-mon[25639]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]': finished 2024-09-21T12:27:46.317 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:46 smithi104 ceph-mon[25639]: osdmap e45: 7 total, 6 up, 7 in 2024-09-21T12:27:46.317 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:46 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T12:27:46.317 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:46 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T12:27:46.317 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:46 smithi104 ceph-mon[25639]: from='osd.6 ' entity='osd.6' 2024-09-21T12:27:46.317 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:46 smithi104 ceph-mon[25639]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T12:27:46.693 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:46 smithi079 ceph-mon[23699]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]': finished 2024-09-21T12:27:46.693 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:46 smithi079 ceph-mon[23699]: osdmap e45: 7 total, 6 up, 7 in 2024-09-21T12:27:46.694 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:46 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T12:27:46.694 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:46 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T12:27:46.694 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:46 smithi079 ceph-mon[23699]: from='osd.6 ' entity='osd.6' 2024-09-21T12:27:46.694 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:46 smithi079 ceph-mon[23699]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T12:27:46.694 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:46 smithi079 ceph-mon[30023]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]': finished 2024-09-21T12:27:46.695 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:46 smithi079 ceph-mon[30023]: osdmap e45: 7 total, 6 up, 7 in 2024-09-21T12:27:46.695 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:46 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T12:27:46.695 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:46 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T12:27:46.695 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:46 smithi079 ceph-mon[30023]: from='osd.6 ' entity='osd.6' 2024-09-21T12:27:46.695 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:46 smithi079 ceph-mon[30023]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T12:27:47.515 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:47 smithi104 ceph-mon[25639]: purged_snaps scrub starts 2024-09-21T12:27:47.515 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:47 smithi104 ceph-mon[25639]: purged_snaps scrub ok 2024-09-21T12:27:47.515 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:47 smithi104 ceph-mon[25639]: osd.6 [v2:172.21.15.104:6816/2903175260,v1:172.21.15.104:6817/2903175260] boot 2024-09-21T12:27:47.515 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:47 smithi104 ceph-mon[25639]: osdmap e46: 7 total, 7 up, 7 in 2024-09-21T12:27:47.515 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:47 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T12:27:47.515 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:47 smithi104 ceph-mon[25639]: osdmap e47: 7 total, 7 up, 7 in 2024-09-21T12:27:47.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:47 smithi079 ceph-mon[23699]: purged_snaps scrub starts 2024-09-21T12:27:47.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:47 smithi079 ceph-mon[23699]: purged_snaps scrub ok 2024-09-21T12:27:47.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:47 smithi079 ceph-mon[23699]: osd.6 [v2:172.21.15.104:6816/2903175260,v1:172.21.15.104:6817/2903175260] boot 2024-09-21T12:27:47.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:47 smithi079 ceph-mon[23699]: osdmap e46: 7 total, 7 up, 7 in 2024-09-21T12:27:47.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:47 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T12:27:47.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:47 smithi079 ceph-mon[23699]: osdmap e47: 7 total, 7 up, 7 in 2024-09-21T12:27:47.692 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:47 smithi079 ceph-mon[30023]: purged_snaps scrub starts 2024-09-21T12:27:47.692 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:47 smithi079 ceph-mon[30023]: purged_snaps scrub ok 2024-09-21T12:27:47.693 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:47 smithi079 ceph-mon[30023]: osd.6 [v2:172.21.15.104:6816/2903175260,v1:172.21.15.104:6817/2903175260] boot 2024-09-21T12:27:47.693 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:47 smithi079 ceph-mon[30023]: osdmap e46: 7 total, 7 up, 7 in 2024-09-21T12:27:47.693 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:47 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T12:27:47.693 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:47 smithi079 ceph-mon[30023]: osdmap e47: 7 total, 7 up, 7 in 2024-09-21T12:27:48.229 INFO:teuthology.orchestra.run.smithi104.stdout: 2024-09-21T12:27:48.247 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 33168022-7814-11ef-baf6-efdc52797490 -- ceph orch daemon add osd smithi104:vg_nvme/lv_1 2024-09-21T12:27:48.441 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:27:48.855 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:48 smithi104 ceph-mon[25639]: osdmap e48: 7 total, 7 up, 7 in 2024-09-21T12:27:48.855 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:48 smithi104 ceph-mon[25639]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-21T12:27:49.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:48 smithi079 ceph-mon[30023]: osdmap e48: 7 total, 7 up, 7 in 2024-09-21T12:27:49.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:48 smithi079 ceph-mon[30023]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-21T12:27:49.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:48 smithi079 ceph-mon[23699]: osdmap e48: 7 total, 7 up, 7 in 2024-09-21T12:27:49.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:48 smithi079 ceph-mon[23699]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-21T12:27:49.827 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:49 smithi104 ceph-mon[25639]: osdmap e49: 7 total, 7 up, 7 in 2024-09-21T12:27:50.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:49 smithi079 ceph-mon[23699]: osdmap e49: 7 total, 7 up, 7 in 2024-09-21T12:27:50.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:49 smithi079 ceph-mon[30023]: osdmap e49: 7 total, 7 up, 7 in 2024-09-21T12:27:51.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:50 smithi104 ceph-mon[25639]: osdmap e50: 7 total, 7 up, 7 in 2024-09-21T12:27:51.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:50 smithi104 ceph-mon[25639]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-21T12:27:51.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:50 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-21T12:27:51.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:50 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-21T12:27:51.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:50 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:27:51.193 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:50 smithi079 ceph-mon[23699]: osdmap e50: 7 total, 7 up, 7 in 2024-09-21T12:27:51.194 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:50 smithi079 ceph-mon[23699]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-21T12:27:51.194 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:50 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-21T12:27:51.194 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:50 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-21T12:27:51.194 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:50 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:27:51.194 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:50 smithi079 ceph-mon[30023]: osdmap e50: 7 total, 7 up, 7 in 2024-09-21T12:27:51.194 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:50 smithi079 ceph-mon[30023]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-21T12:27:51.194 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:50 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-21T12:27:51.194 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:50 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-21T12:27:51.194 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:50 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:27:51.896 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:51 smithi104 ceph-mon[25639]: from='client.24289 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi104:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:27:52.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:51 smithi079 ceph-mon[23699]: from='client.24289 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi104:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:27:52.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:51 smithi079 ceph-mon[30023]: from='client.24289 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi104:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:27:53.094 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:52 smithi104 ceph-mon[25639]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-09-21T12:27:53.094 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:52 smithi104 ceph-mon[25639]: from='client.? 172.21.15.104:0/1498466835' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9003b8d2-e669-42bd-a24d-a556d4db2cf3"}]: dispatch 2024-09-21T12:27:53.094 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:52 smithi104 ceph-mon[25639]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9003b8d2-e669-42bd-a24d-a556d4db2cf3"}]: dispatch 2024-09-21T12:27:53.094 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:52 smithi104 ceph-mon[25639]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9003b8d2-e669-42bd-a24d-a556d4db2cf3"}]': finished 2024-09-21T12:27:53.094 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:52 smithi104 ceph-mon[25639]: osdmap e51: 8 total, 7 up, 8 in 2024-09-21T12:27:53.094 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:52 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T12:27:53.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:52 smithi079 ceph-mon[30023]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-09-21T12:27:53.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:52 smithi079 ceph-mon[30023]: from='client.? 172.21.15.104:0/1498466835' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9003b8d2-e669-42bd-a24d-a556d4db2cf3"}]: dispatch 2024-09-21T12:27:53.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:52 smithi079 ceph-mon[30023]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9003b8d2-e669-42bd-a24d-a556d4db2cf3"}]: dispatch 2024-09-21T12:27:53.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:52 smithi079 ceph-mon[30023]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9003b8d2-e669-42bd-a24d-a556d4db2cf3"}]': finished 2024-09-21T12:27:53.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:52 smithi079 ceph-mon[30023]: osdmap e51: 8 total, 7 up, 8 in 2024-09-21T12:27:53.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:52 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T12:27:53.193 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:52 smithi079 ceph-mon[23699]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail 2024-09-21T12:27:53.193 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:52 smithi079 ceph-mon[23699]: from='client.? 172.21.15.104:0/1498466835' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9003b8d2-e669-42bd-a24d-a556d4db2cf3"}]: dispatch 2024-09-21T12:27:53.193 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:52 smithi079 ceph-mon[23699]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9003b8d2-e669-42bd-a24d-a556d4db2cf3"}]: dispatch 2024-09-21T12:27:53.193 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:52 smithi079 ceph-mon[23699]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9003b8d2-e669-42bd-a24d-a556d4db2cf3"}]': finished 2024-09-21T12:27:53.193 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:52 smithi079 ceph-mon[23699]: osdmap e51: 8 total, 7 up, 8 in 2024-09-21T12:27:53.193 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:52 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T12:27:54.176 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:53 smithi104 ceph-mon[25639]: from='client.? 172.21.15.104:0/4029961156' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-21T12:27:54.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:53 smithi079 ceph-mon[23699]: from='client.? 172.21.15.104:0/4029961156' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-21T12:27:54.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:53 smithi079 ceph-mon[30023]: from='client.? 172.21.15.104:0/4029961156' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-21T12:27:55.176 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:54 smithi104 ceph-mon[25639]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-21T12:27:55.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:54 smithi079 ceph-mon[23699]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-21T12:27:55.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:54 smithi079 ceph-mon[30023]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-21T12:27:56.351 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:56 smithi104 ceph-mon[25639]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-09-21T12:27:56.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:56 smithi079 ceph-mon[23699]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-09-21T12:27:56.442 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:56 smithi079 ceph-mon[30023]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-09-21T12:27:58.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:58 smithi104 ceph-mon[25639]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-09-21T12:27:58.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:58 smithi079 ceph-mon[23699]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-09-21T12:27:58.442 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:58 smithi079 ceph-mon[30023]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-09-21T12:27:59.373 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:59 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-09-21T12:27:59.373 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:59 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:27:59.373 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:27:59 smithi104 ceph-mon[25639]: Deploying daemon osd.7 on smithi104 2024-09-21T12:27:59.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:59 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-09-21T12:27:59.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:59 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:27:59.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:27:59 smithi079 ceph-mon[23699]: Deploying daemon osd.7 on smithi104 2024-09-21T12:27:59.442 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:59 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-09-21T12:27:59.442 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:59 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:27:59.442 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:27:59 smithi079 ceph-mon[30023]: Deploying daemon osd.7 on smithi104 2024-09-21T12:28:00.811 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:00 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:28:00.812 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:00 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:28:00.812 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:00 smithi104 ceph-mon[25639]: Detected new or changed devices on smithi104 2024-09-21T12:28:00.812 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:00 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:28:00.812 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:00 smithi104 ceph-mon[25639]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-21T12:28:00.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:00 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:28:00.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:00 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:28:00.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:00 smithi079 ceph-mon[23699]: Detected new or changed devices on smithi104 2024-09-21T12:28:00.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:00 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:28:00.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:00 smithi079 ceph-mon[23699]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-21T12:28:00.942 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:00 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:28:00.942 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:00 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:28:00.942 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:00 smithi079 ceph-mon[30023]: Detected new or changed devices on smithi104 2024-09-21T12:28:00.943 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:00 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:28:00.943 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:00 smithi079 ceph-mon[30023]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-21T12:28:02.257 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:02 smithi104 ceph-mon[25639]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-21T12:28:02.388 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:02 smithi079 ceph-mon[23699]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-21T12:28:02.388 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:02 smithi079 ceph-mon[30023]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-21T12:28:04.274 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:04 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:28:04.275 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:04 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:28:04.275 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:04 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:28:04.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:04 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:28:04.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:04 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:28:04.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:04 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:28:04.442 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:04 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:28:04.443 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:04 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:28:04.443 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:04 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:28:05.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:05 smithi104 ceph-mon[25639]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-09-21T12:28:05.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:05 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:28:05.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:05 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:28:05.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:05 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:28:05.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:05 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:28:05.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:05 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:28:05.296 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:05 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:28:05.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:05 smithi079 ceph-mon[23699]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-09-21T12:28:05.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:05 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:28:05.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:05 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:28:05.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:05 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:28:05.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:05 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:28:05.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:05 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:28:05.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:05 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:28:05.443 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:05 smithi079 ceph-mon[30023]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-09-21T12:28:05.443 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:05 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:28:05.443 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:05 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:28:05.443 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:05 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:28:05.443 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:05 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:28:05.443 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:05 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:28:05.443 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:05 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:28:06.284 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:06 smithi104 ceph-mon[25639]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-09-21T12:28:06.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:06 smithi079 ceph-mon[23699]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-09-21T12:28:06.442 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:06 smithi079 ceph-mon[30023]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-09-21T12:28:08.113 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:07 smithi104 ceph-mon[25639]: from='osd.7 [v2:172.21.15.104:6824/3612990772,v1:172.21.15.104:6825/3612990772]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-21T12:28:08.113 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:07 smithi104 ceph-mon[25639]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-21T12:28:08.147 INFO:teuthology.orchestra.run.smithi104.stdout:Created osd(s) 7 on host 'smithi104' 2024-09-21T12:28:08.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:07 smithi079 ceph-mon[23699]: from='osd.7 [v2:172.21.15.104:6824/3612990772,v1:172.21.15.104:6825/3612990772]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-21T12:28:08.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:07 smithi079 ceph-mon[23699]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-21T12:28:08.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:07 smithi079 ceph-mon[30023]: from='osd.7 [v2:172.21.15.104:6824/3612990772,v1:172.21.15.104:6825/3612990772]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-21T12:28:08.193 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:07 smithi079 ceph-mon[30023]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-21T12:28:08.804 DEBUG:teuthology.orchestra.run.smithi104:osd.7> sudo journalctl -f -n 0 -u ceph-33168022-7814-11ef-baf6-efdc52797490@osd.7.service 2024-09-21T12:28:08.807 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2024-09-21T12:28:08.807 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 33168022-7814-11ef-baf6-efdc52797490 -- ceph osd stat -f json 2024-09-21T12:28:09.011 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:28:09.131 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:08 smithi079 ceph-mon[23699]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-21T12:28:09.131 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:08 smithi079 ceph-mon[23699]: osdmap e52: 8 total, 7 up, 8 in 2024-09-21T12:28:09.132 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:08 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T12:28:09.132 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:08 smithi079 ceph-mon[23699]: from='osd.7 [v2:172.21.15.104:6824/3612990772,v1:172.21.15.104:6825/3612990772]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-09-21T12:28:09.132 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:08 smithi079 ceph-mon[23699]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-09-21T12:28:09.132 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:08 smithi079 ceph-mon[23699]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-09-21T12:28:09.132 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:08 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:28:09.132 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:08 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:28:09.132 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:08 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-21T12:28:09.132 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:08 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-21T12:28:09.132 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:08 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:28:09.132 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:08 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-21T12:28:09.132 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:08 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-21T12:28:09.132 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:08 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:28:09.132 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:08 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:28:09.132 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:08 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:28:09.133 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:08 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:28:09.133 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:08 smithi079 ceph-mon[30023]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-21T12:28:09.133 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:08 smithi079 ceph-mon[30023]: osdmap e52: 8 total, 7 up, 8 in 2024-09-21T12:28:09.133 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:08 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T12:28:09.133 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:08 smithi079 ceph-mon[30023]: from='osd.7 [v2:172.21.15.104:6824/3612990772,v1:172.21.15.104:6825/3612990772]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-09-21T12:28:09.133 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:08 smithi079 ceph-mon[30023]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-09-21T12:28:09.133 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:08 smithi079 ceph-mon[30023]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-09-21T12:28:09.133 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:08 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:28:09.133 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:08 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:28:09.133 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:08 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-21T12:28:09.133 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:08 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-21T12:28:09.133 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:08 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:28:09.134 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:08 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-21T12:28:09.134 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:08 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-21T12:28:09.134 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:08 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:28:09.134 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:08 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:28:09.134 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:08 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:28:09.134 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:08 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:28:09.177 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 12:28:08 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-7[42127]: 2024-09-21T12:28:08.833+0000 7f552f23e640 -1 osd.7 0 waiting for initial osdmap 2024-09-21T12:28:09.177 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 12:28:08 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-7[42127]: 2024-09-21T12:28:08.843+0000 7f552b067640 -1 osd.7 53 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T12:28:09.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:08 smithi104 ceph-mon[25639]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-21T12:28:09.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:08 smithi104 ceph-mon[25639]: osdmap e52: 8 total, 7 up, 8 in 2024-09-21T12:28:09.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:08 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T12:28:09.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:08 smithi104 ceph-mon[25639]: from='osd.7 [v2:172.21.15.104:6824/3612990772,v1:172.21.15.104:6825/3612990772]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-09-21T12:28:09.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:08 smithi104 ceph-mon[25639]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]: dispatch 2024-09-21T12:28:09.178 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:08 smithi104 ceph-mon[25639]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-09-21T12:28:09.178 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:08 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:28:09.178 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:08 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:28:09.178 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:08 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-21T12:28:09.178 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:08 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-21T12:28:09.178 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:08 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:28:09.178 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:08 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-21T12:28:09.178 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:08 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-21T12:28:09.178 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:08 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:28:09.178 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:08 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:28:09.178 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:08 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:28:09.178 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:08 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:28:09.970 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:09 smithi079 ceph-mon[23699]: Adjusting osd_memory_target on smithi104 to 4016M 2024-09-21T12:28:09.970 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:09 smithi079 ceph-mon[23699]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]': finished 2024-09-21T12:28:09.970 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:09 smithi079 ceph-mon[23699]: osdmap e53: 8 total, 7 up, 8 in 2024-09-21T12:28:09.970 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:09 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T12:28:09.970 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:09 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T12:28:09.970 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:09 smithi079 ceph-mon[30023]: Adjusting osd_memory_target on smithi104 to 4016M 2024-09-21T12:28:09.970 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:09 smithi079 ceph-mon[30023]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]': finished 2024-09-21T12:28:09.971 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:09 smithi079 ceph-mon[30023]: osdmap e53: 8 total, 7 up, 8 in 2024-09-21T12:28:09.971 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:09 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T12:28:09.971 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:09 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T12:28:10.176 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:09 smithi104 ceph-mon[25639]: Adjusting osd_memory_target on smithi104 to 4016M 2024-09-21T12:28:10.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:09 smithi104 ceph-mon[25639]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi104", "root=default"]}]': finished 2024-09-21T12:28:10.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:09 smithi104 ceph-mon[25639]: osdmap e53: 8 total, 7 up, 8 in 2024-09-21T12:28:10.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:09 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T12:28:10.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:09 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T12:28:10.647 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-21T12:28:10.909 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:10 smithi079 ceph-mon[23699]: purged_snaps scrub starts 2024-09-21T12:28:10.909 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:10 smithi079 ceph-mon[23699]: purged_snaps scrub ok 2024-09-21T12:28:10.909 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:10 smithi079 ceph-mon[23699]: osd.7 [v2:172.21.15.104:6824/3612990772,v1:172.21.15.104:6825/3612990772] boot 2024-09-21T12:28:10.909 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:10 smithi079 ceph-mon[23699]: osdmap e54: 8 total, 8 up, 8 in 2024-09-21T12:28:10.910 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:10 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T12:28:10.910 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:10 smithi079 ceph-mon[23699]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:28:10.910 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:10 smithi079 ceph-mon[23699]: from='client.? 172.21.15.79:0/3324598537' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-21T12:28:10.910 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:10 smithi079 ceph-mon[30023]: purged_snaps scrub starts 2024-09-21T12:28:10.910 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:10 smithi079 ceph-mon[30023]: purged_snaps scrub ok 2024-09-21T12:28:10.910 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:10 smithi079 ceph-mon[30023]: osd.7 [v2:172.21.15.104:6824/3612990772,v1:172.21.15.104:6825/3612990772] boot 2024-09-21T12:28:10.910 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:10 smithi079 ceph-mon[30023]: osdmap e54: 8 total, 8 up, 8 in 2024-09-21T12:28:10.910 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:10 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T12:28:10.910 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:10 smithi079 ceph-mon[30023]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:28:10.910 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:10 smithi079 ceph-mon[30023]: from='client.? 172.21.15.79:0/3324598537' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-21T12:28:11.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:10 smithi104 ceph-mon[25639]: purged_snaps scrub starts 2024-09-21T12:28:11.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:10 smithi104 ceph-mon[25639]: purged_snaps scrub ok 2024-09-21T12:28:11.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:10 smithi104 ceph-mon[25639]: osd.7 [v2:172.21.15.104:6824/3612990772,v1:172.21.15.104:6825/3612990772] boot 2024-09-21T12:28:11.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:10 smithi104 ceph-mon[25639]: osdmap e54: 8 total, 8 up, 8 in 2024-09-21T12:28:11.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:10 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T12:28:11.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:10 smithi104 ceph-mon[25639]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:28:11.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:10 smithi104 ceph-mon[25639]: from='client.? 172.21.15.79:0/3324598537' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-21T12:28:11.239 INFO:teuthology.orchestra.run.smithi079.stdout:{"epoch":54,"num_osds":8,"num_up_osds":8,"osd_up_since":1726921689,"num_in_osds":8,"osd_in_since":1726921672,"num_remapped_pgs":1} 2024-09-21T12:28:11.242 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 33168022-7814-11ef-baf6-efdc52797490 -- ceph osd dump --format=json 2024-09-21T12:28:11.434 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:28:12.176 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:11 smithi104 ceph-mon[25639]: osdmap e55: 8 total, 8 up, 8 in 2024-09-21T12:28:12.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:11 smithi079 ceph-mon[23699]: osdmap e55: 8 total, 8 up, 8 in 2024-09-21T12:28:12.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:11 smithi079 ceph-mon[30023]: osdmap e55: 8 total, 8 up, 8 in 2024-09-21T12:28:13.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:12 smithi104 ceph-mon[25639]: osdmap e56: 8 total, 8 up, 8 in 2024-09-21T12:28:13.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:12 smithi104 ceph-mon[25639]: pgmap v169: 1 pgs: 1 remapped+peering; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:28:13.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:12 smithi079 ceph-mon[30023]: osdmap e56: 8 total, 8 up, 8 in 2024-09-21T12:28:13.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:12 smithi079 ceph-mon[30023]: pgmap v169: 1 pgs: 1 remapped+peering; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:28:13.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:12 smithi079 ceph-mon[23699]: osdmap e56: 8 total, 8 up, 8 in 2024-09-21T12:28:13.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:12 smithi079 ceph-mon[23699]: pgmap v169: 1 pgs: 1 remapped+peering; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:28:13.231 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-21T12:28:13.231 INFO:teuthology.orchestra.run.smithi079.stdout:{"epoch":57,"fsid":"33168022-7814-11ef-baf6-efdc52797490","created":"2024-09-21T12:23:21.896565+0000","modified":"2024-09-21T12:28:12.844899+0000","last_up_change":"2024-09-21T12:28:09.822191+0000","last_in_change":"2024-09-21T12:27:52.435732+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T12:26:19.078594+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"aa2c4408-1b95-4d23-9f60-d73b204290db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6802","nonce":846480130},{"type":"v1","addr":"172.21.15.79:6803","nonce":846480130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6804","nonce":846480130},{"type":"v1","addr":"172.21.15.79:6805","nonce":846480130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6808","nonce":846480130},{"type":"v1","addr":"172.21.15.79:6809","nonce":846480130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6806","nonce":846480130},{"type":"v1","addr":"172.21.15.79:6807","nonce":846480130}]},"public_addr":"172.21.15.79:6803/846480130","cluster_addr":"172.21.15.79:6805/846480130","heartbeat_back_addr":"172.21.15.79:6809/846480130","heartbeat_front_addr":"172.21.15.79:6807/846480130","state":["exists","up"]},{"osd":1,"uuid":"a05b4be6-79b2-4dde-bd51-14cafd833c5a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6810","nonce":1764722078},{"type":"v1","addr":"172.21.15.79:6811","nonce":1764722078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6812","nonce":1764722078},{"type":"v1","addr":"172.21.15.79:6813","nonce":1764722078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6816","nonce":1764722078},{"type":"v1","addr":"172.21.15.79:6817","nonce":1764722078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6814","nonce":1764722078},{"type":"v1","addr":"172.21.15.79:6815","nonce":1764722078}]},"public_addr":"172.21.15.79:6811/1764722078","cluster_addr":"172.21.15.79:6813/1764722078","heartbeat_back_addr":"172.21.15.79:6817/1764722078","heartbeat_front_addr":"172.21.15.79:6815/1764722078","state":["exists","up"]},{"osd":2,"uuid":"8b9ddbf7-8264-4e90-bcf0-e63a7f5e7e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6818","nonce":2982011382},{"type":"v1","addr":"172.21.15.79:6819","nonce":2982011382}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6820","nonce":2982011382},{"type":"v1","addr":"172.21.15.79:6821","nonce":2982011382}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6824","nonce":2982011382},{"type":"v1","addr":"172.21.15.79:6825","nonce":2982011382}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6822","nonce":2982011382},{"type":"v1","addr":"172.21.15.79:6823","nonce":2982011382}]},"public_addr":"172.21.15.79:6819/2982011382","cluster_addr":"172.21.15.79:6821/2982011382","heartbeat_back_addr":"172.21.15.79:6825/2982011382","heartbeat_front_addr":"172.21.15.79:6823/2982011382","state":["exists","up"]},{"osd":3,"uuid":"65ac97f0-572a-400f-977b-4c20cdc0fc9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6826","nonce":2894016442},{"type":"v1","addr":"172.21.15.79:6827","nonce":2894016442}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6828","nonce":2894016442},{"type":"v1","addr":"172.21.15.79:6829","nonce":2894016442}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6832","nonce":2894016442},{"type":"v1","addr":"172.21.15.79:6833","nonce":2894016442}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6830","nonce":2894016442},{"type":"v1","addr":"172.21.15.79:6831","nonce":2894016442}]},"public_addr":"172.21.15.79:6827/2894016442","cluster_addr":"172.21.15.79:6829/2894016442","heartbeat_back_addr":"172.21.15.79:6833/2894016442","heartbeat_front_addr":"172.21.15.79:6831/2894016442","state":["exists","up"]},{"osd":4,"uuid":"278156d6-8fd6-4852-9de2-1e11ea7e8917","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":3796154758},{"type":"v1","addr":"172.21.15.104:6801","nonce":3796154758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":3796154758},{"type":"v1","addr":"172.21.15.104:6803","nonce":3796154758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":3796154758},{"type":"v1","addr":"172.21.15.104:6807","nonce":3796154758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":3796154758},{"type":"v1","addr":"172.21.15.104:6805","nonce":3796154758}]},"public_addr":"172.21.15.104:6801/3796154758","cluster_addr":"172.21.15.104:6803/3796154758","heartbeat_back_addr":"172.21.15.104:6807/3796154758","heartbeat_front_addr":"172.21.15.104:6805/3796154758","state":["exists","up"]},{"osd":5,"uuid":"5435eabf-f4d1-4c86-b334-fa3acc9162c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":189336043},{"type":"v1","addr":"172.21.15.104:6809","nonce":189336043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":189336043},{"type":"v1","addr":"172.21.15.104:6811","nonce":189336043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":189336043},{"type":"v1","addr":"172.21.15.104:6815","nonce":189336043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":189336043},{"type":"v1","addr":"172.21.15.104:6813","nonce":189336043}]},"public_addr":"172.21.15.104:6809/189336043","cluster_addr":"172.21.15.104:6811/189336043","heartbeat_back_addr":"172.21.15.104:6815/189336043","heartbeat_front_addr":"172.21.15.104:6813/189336043","state":["exists","up"]},{"osd":6,"uuid":"df36e40d-a08e-4b0f-8774-4862209d0ba0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2903175260},{"type":"v1","addr":"172.21.15.104:6817","nonce":2903175260}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2903175260},{"type":"v1","addr":"172.21.15.104:6819","nonce":2903175260}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2903175260},{"type":"v1","addr":"172.21.15.104:6823","nonce":2903175260}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2903175260},{"type":"v1","addr":"172.21.15.104:6821","nonce":2903175260}]},"public_addr":"172.21.15.104:6817/2903175260","cluster_addr":"172.21.15.104:6819/2903175260","heartbeat_back_addr":"172.21.15.104:6823/2903175260","heartbeat_front_addr":"172.21.15.104:6821/2903175260","state":["exists","up"]},{"osd":7,"uuid":"9003b8d2-e669-42bd-a24d-a556d4db2cf3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":3612990772},{"type":"v1","addr":"172.21.15.104:6825","nonce":3612990772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":3612990772},{"type":"v1","addr":"172.21.15.104:6827","nonce":3612990772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":3612990772},{"type":"v1","addr":"172.21.15.104:6831","nonce":3612990772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":3612990772},{"type":"v1","addr":"172.21.15.104:6829","nonce":3612990772}]},"public_addr":"172.21.15.104:6825/3612990772","cluster_addr":"172.21.15.104:6827/3612990772","heartbeat_back_addr":"172.21.15.104:6831/3612990772","heartbeat_front_addr":"172.21.15.104:6829/3612990772","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T12:25:33.048636+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T12:25:54.292231+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T12:26:15.498328+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T12:26:37.596296+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T12:26:58.051100+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T12:27:21.613879+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T12:27:44.820201+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T12:28:08.730029+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.79:0/2503007178":"2024-09-22T12:24:10.760847+0000","172.21.15.79:6801/3479175059":"2024-09-22T12:24:10.760847+0000","172.21.15.79:6800/3479175059":"2024-09-22T12:24:10.760847+0000","172.21.15.79:6801/3138719122":"2024-09-22T12:23:45.405786+0000","172.21.15.79:0/1561731591":"2024-09-22T12:24:10.760847+0000","172.21.15.79:6800/3138719122":"2024-09-22T12:23:45.405786+0000","172.21.15.79:0/3967018586":"2024-09-22T12:23:45.405786+0000","172.21.15.79:0/1207296335":"2024-09-22T12:23:45.405786+0000","172.21.15.79:0/3564175523":"2024-09-22T12:24:10.760847+0000","172.21.15.79:0/4127214407":"2024-09-22T12:23:45.405786+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T12:28:13.813 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-21T12:26:19.078594+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-21T12:28:13.813 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 33168022-7814-11ef-baf6-efdc52797490 -- ceph osd pool get .mgr pg_num 2024-09-21T12:28:14.014 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:28:14.134 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:13 smithi079 ceph-mon[23699]: osdmap e57: 8 total, 8 up, 8 in 2024-09-21T12:28:14.134 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:13 smithi079 ceph-mon[23699]: from='client.? 172.21.15.79:0/273140612' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-21T12:28:14.135 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:13 smithi079 ceph-mon[30023]: osdmap e57: 8 total, 8 up, 8 in 2024-09-21T12:28:14.135 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:13 smithi079 ceph-mon[30023]: from='client.? 172.21.15.79:0/273140612' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-21T12:28:14.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:13 smithi104 ceph-mon[25639]: osdmap e57: 8 total, 8 up, 8 in 2024-09-21T12:28:14.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:13 smithi104 ceph-mon[25639]: from='client.? 172.21.15.79:0/273140612' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-21T12:28:15.164 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:14 smithi079 ceph-mon[23699]: osdmap e58: 8 total, 8 up, 8 in 2024-09-21T12:28:15.165 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:14 smithi079 ceph-mon[23699]: pgmap v172: 1 pgs: 1 remapped+peering; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:28:15.165 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:14 smithi079 ceph-mon[30023]: osdmap e58: 8 total, 8 up, 8 in 2024-09-21T12:28:15.165 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:14 smithi079 ceph-mon[30023]: pgmap v172: 1 pgs: 1 remapped+peering; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:28:15.176 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:14 smithi104 ceph-mon[25639]: osdmap e58: 8 total, 8 up, 8 in 2024-09-21T12:28:15.176 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:14 smithi104 ceph-mon[25639]: pgmap v172: 1 pgs: 1 remapped+peering; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:28:15.816 INFO:teuthology.orchestra.run.smithi079.stdout:pg_num: 1 2024-09-21T12:28:16.161 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:15 smithi079 ceph-mon[23699]: from='client.? 172.21.15.79:0/3694491853' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-21T12:28:16.162 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:15 smithi079 ceph-mon[30023]: from='client.? 172.21.15.79:0/3694491853' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-21T12:28:16.176 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:15 smithi104 ceph-mon[25639]: from='client.? 172.21.15.79:0/3694491853' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-21T12:28:16.484 INFO:tasks.cephadm:Adding prometheus.a on smithi104 2024-09-21T12:28:16.484 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 33168022-7814-11ef-baf6-efdc52797490 -- ceph orch apply prometheus '1;smithi104=a' 2024-09-21T12:28:16.713 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:28:17.135 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:16 smithi104 ceph-mon[25639]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 111 KiB/s, 0 objects/s recovering 2024-09-21T12:28:17.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:16 smithi079 ceph-mon[30023]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 111 KiB/s, 0 objects/s recovering 2024-09-21T12:28:17.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:16 smithi079 ceph-mon[23699]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 111 KiB/s, 0 objects/s recovering 2024-09-21T12:28:18.400 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:18 smithi104 ceph-mon[25639]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 93 KiB/s, 0 objects/s recovering 2024-09-21T12:28:18.442 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:18 smithi079 ceph-mon[30023]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 93 KiB/s, 0 objects/s recovering 2024-09-21T12:28:18.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:18 smithi079 ceph-mon[23699]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 93 KiB/s, 0 objects/s recovering 2024-09-21T12:28:18.600 INFO:teuthology.orchestra.run.smithi104.stdout:Scheduled prometheus update... 2024-09-21T12:28:19.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:19 smithi104 ceph-mon[25639]: from='client.24322 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi104=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:28:19.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:19 smithi104 ceph-mon[25639]: Saving service prometheus spec with placement smithi104=a;count:1 2024-09-21T12:28:19.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:19 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:28:19.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:19 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:28:19.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:19 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:28:19.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:19 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:28:19.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:19 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:28:19.678 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:19 smithi104 ceph-mon[25639]: Deploying daemon prometheus.a on smithi104 2024-09-21T12:28:19.812 DEBUG:teuthology.orchestra.run.smithi104:prometheus.a> sudo journalctl -f -n 0 -u ceph-33168022-7814-11ef-baf6-efdc52797490@prometheus.a.service 2024-09-21T12:28:19.814 INFO:tasks.cephadm:Adding node-exporter.a on smithi079 2024-09-21T12:28:19.815 INFO:tasks.cephadm:Adding node-exporter.b on smithi104 2024-09-21T12:28:19.815 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 33168022-7814-11ef-baf6-efdc52797490 -- ceph orch apply node-exporter '2;smithi079=a;smithi104=b' 2024-09-21T12:28:19.942 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:19 smithi079 ceph-mon[30023]: from='client.24322 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi104=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:28:19.942 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:19 smithi079 ceph-mon[30023]: Saving service prometheus spec with placement smithi104=a;count:1 2024-09-21T12:28:19.942 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:19 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:28:19.942 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:19 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:28:19.942 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:19 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:28:19.942 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:19 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:28:19.942 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:19 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:28:19.943 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:19 smithi079 ceph-mon[30023]: Deploying daemon prometheus.a on smithi104 2024-09-21T12:28:19.943 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:19 smithi079 ceph-mon[23699]: from='client.24322 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi104=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:28:19.943 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:19 smithi079 ceph-mon[23699]: Saving service prometheus spec with placement smithi104=a;count:1 2024-09-21T12:28:19.943 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:19 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:28:19.943 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:19 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:28:19.943 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:19 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:28:19.943 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:19 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:28:19.943 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:19 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:28:19.943 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:19 smithi079 ceph-mon[23699]: Deploying daemon prometheus.a on smithi104 2024-09-21T12:28:20.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:20 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:28:20.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:20 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:28:20.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:20 smithi104 ceph-mon[25639]: Detected new or changed devices on smithi104 2024-09-21T12:28:20.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:20 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:28:20.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:20 smithi104 ceph-mon[25639]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-21T12:28:20.941 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:20 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:28:20.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:20 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:28:20.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:20 smithi079 ceph-mon[23699]: Detected new or changed devices on smithi104 2024-09-21T12:28:20.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:20 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:28:20.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:20 smithi079 ceph-mon[23699]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-21T12:28:20.942 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:20 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:28:20.942 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:20 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:28:20.942 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:20 smithi079 ceph-mon[30023]: Detected new or changed devices on smithi104 2024-09-21T12:28:20.942 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:20 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:28:20.942 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:20 smithi079 ceph-mon[30023]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-21T12:28:21.434 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:28:22.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:21 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:28:22.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:21 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:28:22.077 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:21 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:28:22.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:21 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:28:22.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:21 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:28:22.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:21 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:28:22.193 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:21 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:28:22.193 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:21 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:28:22.193 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:21 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:28:23.147 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:22 smithi079 ceph-mon[23699]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 63 KiB/s, 0 objects/s recovering 2024-09-21T12:28:23.148 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:22 smithi079 ceph-mon[30023]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 63 KiB/s, 0 objects/s recovering 2024-09-21T12:28:23.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:22 smithi104 ceph-mon[25639]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 63 KiB/s, 0 objects/s recovering 2024-09-21T12:28:23.869 INFO:teuthology.orchestra.run.smithi104.stdout:Scheduled node-exporter update... 2024-09-21T12:28:24.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:24 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:28:24.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:24 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:28:24.428 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:24 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:28:24.428 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:24 smithi104 ceph-mon[25639]: from='client.24328 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi079=a;smithi104=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:28:24.428 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:24 smithi104 ceph-mon[25639]: Saving service node-exporter spec with placement smithi079=a;smithi104=b;count:2 2024-09-21T12:28:24.428 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:24 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:28:24.428 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:24 smithi104 ceph-mon[25639]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 57 KiB/s, 0 objects/s recovering 2024-09-21T12:28:24.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:24 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:28:24.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:24 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:28:24.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:24 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:28:24.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:24 smithi079 ceph-mon[23699]: from='client.24328 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi079=a;smithi104=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:28:24.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:24 smithi079 ceph-mon[23699]: Saving service node-exporter spec with placement smithi079=a;smithi104=b;count:2 2024-09-21T12:28:24.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:24 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:28:24.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:24 smithi079 ceph-mon[23699]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 57 KiB/s, 0 objects/s recovering 2024-09-21T12:28:24.443 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:24 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:28:24.443 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:24 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:28:24.443 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:24 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:28:24.443 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:24 smithi079 ceph-mon[30023]: from='client.24328 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi079=a;smithi104=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:28:24.443 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:24 smithi079 ceph-mon[30023]: Saving service node-exporter spec with placement smithi079=a;smithi104=b;count:2 2024-09-21T12:28:24.443 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:24 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:28:24.443 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:24 smithi079 ceph-mon[30023]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 57 KiB/s, 0 objects/s recovering 2024-09-21T12:28:25.867 DEBUG:teuthology.orchestra.run.smithi079:node-exporter.a> sudo journalctl -f -n 0 -u ceph-33168022-7814-11ef-baf6-efdc52797490@node-exporter.a.service 2024-09-21T12:28:25.871 DEBUG:teuthology.orchestra.run.smithi104:node-exporter.b> sudo journalctl -f -n 0 -u ceph-33168022-7814-11ef-baf6-efdc52797490@node-exporter.b.service 2024-09-21T12:28:25.873 INFO:tasks.cephadm:Adding alertmanager.a on smithi079 2024-09-21T12:28:25.873 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 33168022-7814-11ef-baf6-efdc52797490 -- ceph orch apply alertmanager '1;smithi079=a' 2024-09-21T12:28:26.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:26 smithi079 ceph-mon[23699]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-21T12:28:26.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:26 smithi079 ceph-mon[30023]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-21T12:28:26.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:26 smithi104 ceph-mon[25639]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-09-21T12:28:26.682 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:28:28.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:28 smithi104 ceph-mon[25639]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:28:28.441 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:28 smithi079 ceph-mon[23699]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:28:28.442 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:28 smithi079 ceph-mon[30023]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:28:30.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:30 smithi104 ceph-mon[25639]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:28:30.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:30 smithi079 ceph-mon[23699]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:28:30.442 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:30 smithi079 ceph-mon[30023]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:28:30.821 INFO:teuthology.orchestra.run.smithi104.stdout:Scheduled alertmanager update... 2024-09-21T12:28:32.049 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:31 smithi104 ceph-mon[25639]: from='client.24334 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi079=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:28:32.049 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:31 smithi104 ceph-mon[25639]: Saving service alertmanager spec with placement smithi079=a;count:1 2024-09-21T12:28:32.049 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:31 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:28:32.073 DEBUG:teuthology.orchestra.run.smithi079:alertmanager.a> sudo journalctl -f -n 0 -u ceph-33168022-7814-11ef-baf6-efdc52797490@alertmanager.a.service 2024-09-21T12:28:32.077 INFO:tasks.cephadm:Adding grafana.a on smithi104 2024-09-21T12:28:32.077 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 33168022-7814-11ef-baf6-efdc52797490 -- ceph orch apply grafana '1;smithi104=a' 2024-09-21T12:28:32.100 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:31 smithi079 ceph-mon[23699]: from='client.24334 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi079=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:28:32.101 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:31 smithi079 ceph-mon[23699]: Saving service alertmanager spec with placement smithi079=a;count:1 2024-09-21T12:28:32.101 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:31 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:28:32.101 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:31 smithi079 ceph-mon[30023]: from='client.24334 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi079=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:28:32.101 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:31 smithi079 ceph-mon[30023]: Saving service alertmanager spec with placement smithi079=a;count:1 2024-09-21T12:28:32.101 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:31 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:28:32.760 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:28:32 smithi104 systemd[1]: Started Ceph prometheus.a for 33168022-7814-11ef-baf6-efdc52797490. 2024-09-21T12:28:33.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:32 smithi104 ceph-mon[25639]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:28:33.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:32 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:28:33.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:32 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:28:33.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:32 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:28:33.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:32 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-09-21T12:28:33.114 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:28:33.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:32 smithi079 ceph-mon[23699]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:28:33.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:32 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:28:33.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:32 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:28:33.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:32 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:28:33.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:32 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-09-21T12:28:33.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:32 smithi079 ceph-mon[30023]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:28:33.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:32 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:28:33.193 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:32 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:28:33.193 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:32 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' 2024-09-21T12:28:33.193 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:32 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-09-21T12:28:33.927 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:28:33 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[27828]: ignoring --setuser ceph since I am not root 2024-09-21T12:28:33.927 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:28:33 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[27828]: ignoring --setgroup ceph since I am not root 2024-09-21T12:28:33.927 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:28:33 smithi104 ceph-mgr[27832]: -- 172.21.15.104:0/9154669 <== mon.1 v2:172.21.15.104:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x560b0b60ad00 con 0x560b0b5df400 2024-09-21T12:28:33.927 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:28:33 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[27828]: 2024-09-21T12:28:33.774+0000 7f0e2e67c040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-21T12:28:33.927 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:28:33 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[27828]: 2024-09-21T12:28:33.830+0000 7f0e2e67c040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-21T12:28:33.942 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:28:33 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: ignoring --setuser ceph since I am not root 2024-09-21T12:28:33.942 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:28:33 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: ignoring --setgroup ceph since I am not root 2024-09-21T12:28:33.942 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:28:33 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: 2024-09-21T12:28:33.767+0000 7f0413576040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-21T12:28:33.942 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:28:33 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: 2024-09-21T12:28:33.822+0000 7f0413576040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-21T12:28:34.317 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:28:33 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: 2024-09-21T12:28:33.968+0000 7f0413576040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-21T12:28:34.317 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:28:34 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: 2024-09-21T12:28:34.033+0000 7f0413576040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-21T12:28:34.317 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:28:34 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: 2024-09-21T12:28:34.176+0000 7f0413576040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-21T12:28:34.335 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:28:33 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[27828]: 2024-09-21T12:28:33.980+0000 7f0e2e67c040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-21T12:28:34.335 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:28:34 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[27828]: 2024-09-21T12:28:34.045+0000 7f0e2e67c040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-21T12:28:34.335 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:28:34 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[27828]: 2024-09-21T12:28:34.191+0000 7f0e2e67c040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-21T12:28:34.335 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:28:34 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[27828]: 2024-09-21T12:28:34.333+0000 7f0e2e67c040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-21T12:28:34.618 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:28:34 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: 2024-09-21T12:28:34.315+0000 7f0413576040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-21T12:28:34.898 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:34 smithi104 ceph-mon[25639]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-09-21T12:28:34.898 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:34 smithi104 ceph-mon[25639]: mgrmap e16: y(active, since 4m), standbys: x 2024-09-21T12:28:34.899 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:28:34 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[27828]: 2024-09-21T12:28:34.651+0000 7f0e2e67c040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-21T12:28:34.899 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:28:34 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[27828]: 2024-09-21T12:28:34.712+0000 7f0e2e67c040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-21T12:28:34.899 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:28:34 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[27828]: 2024-09-21T12:28:34.896+0000 7f0e2e67c040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-21T12:28:34.941 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:34 smithi079 ceph-mon[23699]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-09-21T12:28:34.941 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:34 smithi079 ceph-mon[23699]: mgrmap e16: y(active, since 4m), standbys: x 2024-09-21T12:28:34.942 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:34 smithi079 ceph-mon[30023]: from='mgr.14150 172.21.15.79:0/2713576905' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-09-21T12:28:34.942 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:34 smithi079 ceph-mon[30023]: mgrmap e16: y(active, since 4m), standbys: x 2024-09-21T12:28:34.942 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:28:34 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: 2024-09-21T12:28:34.616+0000 7f0413576040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-21T12:28:34.942 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:28:34 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: 2024-09-21T12:28:34.677+0000 7f0413576040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-21T12:28:34.942 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:28:34 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: 2024-09-21T12:28:34.859+0000 7f0413576040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-21T12:28:36.177 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:28:35 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[27828]: 2024-09-21T12:28:35.793+0000 7f0e2e67c040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-21T12:28:36.177 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:28:35 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[27828]: 2024-09-21T12:28:35.858+0000 7f0e2e67c040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-21T12:28:36.191 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:28:35 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: 2024-09-21T12:28:35.722+0000 7f0413576040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-21T12:28:36.192 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:28:35 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: 2024-09-21T12:28:35.786+0000 7f0413576040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-21T12:28:36.743 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:28:36 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: 2024-09-21T12:28:36.488+0000 7f0413576040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-21T12:28:36.743 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:28:36 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: 2024-09-21T12:28:36.601+0000 7f0413576040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-21T12:28:36.821 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:28:36 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[27828]: 2024-09-21T12:28:36.565+0000 7f0e2e67c040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-21T12:28:36.821 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:28:36 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[27828]: 2024-09-21T12:28:36.679+0000 7f0e2e67c040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-21T12:28:37.015 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:28:36 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: 2024-09-21T12:28:36.741+0000 7f0413576040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-21T12:28:37.015 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:28:36 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: 2024-09-21T12:28:36.802+0000 7f0413576040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-21T12:28:37.015 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:28:36 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: 2024-09-21T12:28:36.946+0000 7f0413576040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-21T12:28:37.015 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:28:37 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: 2024-09-21T12:28:37.013+0000 7f0413576040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-21T12:28:37.092 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:28:36 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[27828]: 2024-09-21T12:28:36.819+0000 7f0e2e67c040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-21T12:28:37.092 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:28:36 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[27828]: 2024-09-21T12:28:36.879+0000 7f0e2e67c040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-21T12:28:37.092 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:28:37 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[27828]: 2024-09-21T12:28:37.023+0000 7f0e2e67c040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-21T12:28:37.092 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:28:37 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[27828]: 2024-09-21T12:28:37.090+0000 7f0e2e67c040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-21T12:28:37.291 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:28:37 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: 2024-09-21T12:28:37.072+0000 7f0413576040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-21T12:28:37.291 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:28:37 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: 2024-09-21T12:28:37.225+0000 7f0413576040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-21T12:28:37.291 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:28:37 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: 2024-09-21T12:28:37.289+0000 7f0413576040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-21T12:28:37.366 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:28:37 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[27828]: 2024-09-21T12:28:37.149+0000 7f0e2e67c040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-21T12:28:37.367 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:28:37 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[27828]: 2024-09-21T12:28:37.301+0000 7f0e2e67c040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-21T12:28:37.367 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:28:37 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[27828]: 2024-09-21T12:28:37.364+0000 7f0e2e67c040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-21T12:28:37.554 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:28:37 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: 2024-09-21T12:28:37.354+0000 7f0413576040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-21T12:28:37.554 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:28:37 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: 2024-09-21T12:28:37.435+0000 7f0413576040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-21T12:28:37.554 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:28:37 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: 2024-09-21T12:28:37.494+0000 7f0413576040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-21T12:28:37.554 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:28:37 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: 2024-09-21T12:28:37.552+0000 7f0413576040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-21T12:28:37.629 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:28:37 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[27828]: 2024-09-21T12:28:37.431+0000 7f0e2e67c040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-21T12:28:37.629 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:28:37 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[27828]: 2024-09-21T12:28:37.511+0000 7f0e2e67c040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-21T12:28:37.629 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:28:37 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[27828]: 2024-09-21T12:28:37.570+0000 7f0e2e67c040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-21T12:28:37.926 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:28:37 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[27828]: 2024-09-21T12:28:37.627+0000 7f0e2e67c040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-21T12:28:38.677 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:28:38 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[27828]: 2024-09-21T12:28:38.402+0000 7f0e2e67c040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-21T12:28:38.677 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:28:38 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[27828]: [21/Sep/2024:12:28:38] ENGINE Bus STARTING 2024-09-21T12:28:38.677 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:28:38 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[27828]: CherryPy Checker: 2024-09-21T12:28:38.677 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:28:38 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[27828]: The Application mounted at '' has an empty config. 2024-09-21T12:28:38.677 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:28:38 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[27828]: 2024-09-21T12:28:38.677 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:28:38 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[27828]: [21/Sep/2024:12:28:38] ENGINE Serving on http://:::9283 2024-09-21T12:28:38.677 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:28:38 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[27828]: [21/Sep/2024:12:28:38] ENGINE Bus STARTED 2024-09-21T12:28:38.678 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:38 smithi104 ceph-mon[25639]: Active manager daemon y restarted 2024-09-21T12:28:38.678 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:38 smithi104 ceph-mon[25639]: Activating manager daemon y 2024-09-21T12:28:38.678 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:38 smithi104 ceph-mon[25639]: osdmap e59: 8 total, 8 up, 8 in 2024-09-21T12:28:38.678 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:38 smithi104 ceph-mon[25639]: mgrmap e17: y(active, starting, since 0.00648144s), standbys: x 2024-09-21T12:28:38.678 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:38 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T12:28:38.678 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:38 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T12:28:38.678 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:38 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T12:28:38.678 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:38 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-21T12:28:38.678 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:38 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-21T12:28:38.678 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:38 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T12:28:38.678 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:38 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T12:28:38.678 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:38 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T12:28:38.679 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:38 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T12:28:38.679 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:38 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T12:28:38.679 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:38 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T12:28:38.679 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:38 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T12:28:38.679 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:38 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T12:28:38.679 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:38 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-21T12:28:38.679 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:38 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-21T12:28:38.679 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:38 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-21T12:28:38.679 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:38 smithi104 ceph-mon[25639]: Manager daemon y is now available 2024-09-21T12:28:38.679 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:38 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:28:38.679 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:38 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:28:38.679 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:38 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi104", "name": "osd_memory_target"}]: dispatch 2024-09-21T12:28:38.679 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:38 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi079", "name": "osd_memory_target"}]: dispatch 2024-09-21T12:28:38.679 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:38 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi104", "name": "osd_memory_target"}]': finished 2024-09-21T12:28:38.679 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:38 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi079", "name": "osd_memory_target"}]': finished 2024-09-21T12:28:38.682 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:38 smithi079 ceph-mon[23699]: Active manager daemon y restarted 2024-09-21T12:28:38.682 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:38 smithi079 ceph-mon[23699]: Activating manager daemon y 2024-09-21T12:28:38.682 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:38 smithi079 ceph-mon[23699]: osdmap e59: 8 total, 8 up, 8 in 2024-09-21T12:28:38.682 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:38 smithi079 ceph-mon[23699]: mgrmap e17: y(active, starting, since 0.00648144s), standbys: x 2024-09-21T12:28:38.683 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:38 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T12:28:38.683 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:38 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T12:28:38.683 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:38 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T12:28:38.683 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:38 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-21T12:28:38.683 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:38 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-21T12:28:38.683 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:38 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T12:28:38.683 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:38 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T12:28:38.683 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:38 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T12:28:38.683 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:38 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T12:28:38.683 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:38 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T12:28:38.683 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:38 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T12:28:38.683 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:38 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T12:28:38.683 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:38 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T12:28:38.683 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:38 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-21T12:28:38.683 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:38 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-21T12:28:38.683 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:38 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-21T12:28:38.684 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:38 smithi079 ceph-mon[23699]: Manager daemon y is now available 2024-09-21T12:28:38.684 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:38 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:28:38.684 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:38 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:28:38.684 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:38 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi104", "name": "osd_memory_target"}]: dispatch 2024-09-21T12:28:38.684 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:38 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi079", "name": "osd_memory_target"}]: dispatch 2024-09-21T12:28:38.684 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:38 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi104", "name": "osd_memory_target"}]': finished 2024-09-21T12:28:38.684 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:38 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi079", "name": "osd_memory_target"}]': finished 2024-09-21T12:28:38.684 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:28:38 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: 2024-09-21T12:28:38.319+0000 7f0413576040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-21T12:28:38.684 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:28:38 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: [21/Sep/2024:12:28:38] ENGINE Bus STARTING 2024-09-21T12:28:38.684 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:28:38 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: CherryPy Checker: 2024-09-21T12:28:38.684 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:28:38 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: The Application mounted at '' has an empty config. 2024-09-21T12:28:38.685 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:28:38 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: 2024-09-21T12:28:38.685 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:28:38 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: [21/Sep/2024:12:28:38] ENGINE Serving on http://:::9283 2024-09-21T12:28:38.685 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:28:38 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: [21/Sep/2024:12:28:38] ENGINE Bus STARTED 2024-09-21T12:28:38.685 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:38 smithi079 ceph-mon[30023]: Active manager daemon y restarted 2024-09-21T12:28:38.685 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:38 smithi079 ceph-mon[30023]: Activating manager daemon y 2024-09-21T12:28:38.685 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:38 smithi079 ceph-mon[30023]: osdmap e59: 8 total, 8 up, 8 in 2024-09-21T12:28:38.685 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:38 smithi079 ceph-mon[30023]: mgrmap e17: y(active, starting, since 0.00648144s), standbys: x 2024-09-21T12:28:38.685 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:38 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T12:28:38.685 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:38 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T12:28:38.685 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:38 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T12:28:38.685 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:38 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-21T12:28:38.685 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:38 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-21T12:28:38.685 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:38 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T12:28:38.686 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:38 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T12:28:38.686 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:38 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T12:28:38.686 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:38 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T12:28:38.686 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:38 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T12:28:38.686 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:38 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T12:28:38.686 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:38 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T12:28:38.686 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:38 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T12:28:38.686 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:38 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-21T12:28:38.686 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:38 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-21T12:28:38.686 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:38 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-21T12:28:38.686 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:38 smithi079 ceph-mon[30023]: Manager daemon y is now available 2024-09-21T12:28:38.686 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:38 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:28:38.686 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:38 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:28:38.686 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:38 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi104", "name": "osd_memory_target"}]: dispatch 2024-09-21T12:28:38.686 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:38 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi079", "name": "osd_memory_target"}]: dispatch 2024-09-21T12:28:38.687 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:38 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi104", "name": "osd_memory_target"}]': finished 2024-09-21T12:28:38.687 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:38 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi079", "name": "osd_memory_target"}]': finished 2024-09-21T12:28:39.348 INFO:teuthology.orchestra.run.smithi104.stdout:Scheduled grafana update... 2024-09-21T12:28:39.672 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:39 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:28:39.672 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:39 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:28:39.672 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:39 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:28:39.672 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:39 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T12:28:39.672 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:39 smithi104 ceph-mon[25639]: Updating smithi079:/etc/ceph/ceph.conf 2024-09-21T12:28:39.672 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:39 smithi104 ceph-mon[25639]: Updating smithi104:/etc/ceph/ceph.conf 2024-09-21T12:28:39.673 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:39 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-21T12:28:39.673 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:39 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-21T12:28:39.673 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:39 smithi104 ceph-mon[25639]: Standby manager daemon x restarted 2024-09-21T12:28:39.673 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:39 smithi104 ceph-mon[25639]: Standby manager daemon x started 2024-09-21T12:28:39.673 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:39 smithi104 ceph-mon[25639]: from='mgr.? 172.21.15.104:0/4002326613' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-21T12:28:39.673 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:39 smithi104 ceph-mon[25639]: from='mgr.? 172.21.15.104:0/4002326613' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-21T12:28:39.673 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:39 smithi104 ceph-mon[25639]: from='mgr.? 172.21.15.104:0/4002326613' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-21T12:28:39.673 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:39 smithi104 ceph-mon[25639]: from='mgr.? 172.21.15.104:0/4002326613' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-21T12:28:39.673 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:39 smithi104 ceph-mon[25639]: mgrmap e18: y(active, since 1.01232s), standbys: x 2024-09-21T12:28:39.673 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:39 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:28:39.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:39 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:28:39.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:39 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:28:39.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:39 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:28:39.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:39 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T12:28:39.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:39 smithi079 ceph-mon[23699]: Updating smithi079:/etc/ceph/ceph.conf 2024-09-21T12:28:39.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:39 smithi079 ceph-mon[23699]: Updating smithi104:/etc/ceph/ceph.conf 2024-09-21T12:28:39.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:39 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-21T12:28:39.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:39 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-21T12:28:39.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:39 smithi079 ceph-mon[23699]: Standby manager daemon x restarted 2024-09-21T12:28:39.693 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:39 smithi079 ceph-mon[23699]: Standby manager daemon x started 2024-09-21T12:28:39.693 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:39 smithi079 ceph-mon[23699]: from='mgr.? 172.21.15.104:0/4002326613' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-21T12:28:39.693 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:39 smithi079 ceph-mon[23699]: from='mgr.? 172.21.15.104:0/4002326613' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-21T12:28:39.693 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:39 smithi079 ceph-mon[23699]: from='mgr.? 172.21.15.104:0/4002326613' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-21T12:28:39.693 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:39 smithi079 ceph-mon[23699]: from='mgr.? 172.21.15.104:0/4002326613' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-21T12:28:39.693 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:39 smithi079 ceph-mon[23699]: mgrmap e18: y(active, since 1.01232s), standbys: x 2024-09-21T12:28:39.693 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:39 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:28:39.693 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:39 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:28:39.693 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:39 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:28:39.693 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:39 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:28:39.693 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:39 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T12:28:39.693 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:39 smithi079 ceph-mon[30023]: Updating smithi079:/etc/ceph/ceph.conf 2024-09-21T12:28:39.694 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:39 smithi079 ceph-mon[30023]: Updating smithi104:/etc/ceph/ceph.conf 2024-09-21T12:28:39.694 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:39 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-21T12:28:39.694 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:39 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-21T12:28:39.694 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:39 smithi079 ceph-mon[30023]: Standby manager daemon x restarted 2024-09-21T12:28:39.694 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:39 smithi079 ceph-mon[30023]: Standby manager daemon x started 2024-09-21T12:28:39.694 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:39 smithi079 ceph-mon[30023]: from='mgr.? 172.21.15.104:0/4002326613' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-21T12:28:39.694 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:39 smithi079 ceph-mon[30023]: from='mgr.? 172.21.15.104:0/4002326613' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-21T12:28:39.694 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:39 smithi079 ceph-mon[30023]: from='mgr.? 172.21.15.104:0/4002326613' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-21T12:28:39.694 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:39 smithi079 ceph-mon[30023]: from='mgr.? 172.21.15.104:0/4002326613' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-21T12:28:39.694 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:39 smithi079 ceph-mon[30023]: mgrmap e18: y(active, since 1.01232s), standbys: x 2024-09-21T12:28:39.694 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:39 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:28:39.694 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:28:39 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: 2024-09-21T12:28:39.584+0000 7f03712b0640 -1 log_channel(cephadm) log [ERR] : [21/Sep/2024:12:28:39] ENGINE Error in HTTPServer.serve 2024-09-21T12:28:39.694 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:28:39 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: Traceback (most recent call last): 2024-09-21T12:28:39.695 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:28:39 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T12:28:39.695 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:28:39 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: self._connections.run(self.expiration_interval) 2024-09-21T12:28:39.695 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:28:39 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T12:28:39.695 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:28:39 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: self._run(expiration_interval) 2024-09-21T12:28:39.695 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:28:39 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T12:28:39.695 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:28:39 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T12:28:39.695 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:28:39 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T12:28:39.695 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:28:39 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T12:28:39.695 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:28:39 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T12:28:39.695 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:28:39 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: s = self.context.wrap_socket( 2024-09-21T12:28:39.695 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:28:39 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T12:28:39.695 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:28:39 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: return self.sslsocket_class._create( 2024-09-21T12:28:39.695 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:28:39 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T12:28:39.695 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:28:39 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: self.do_handshake() 2024-09-21T12:28:39.696 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:28:39 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T12:28:39.696 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:28:39 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: self._sslobj.do_handshake() 2024-09-21T12:28:39.696 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:28:39 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T12:28:39.696 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:28:39 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: 2024-09-21T12:28:40.599 DEBUG:teuthology.orchestra.run.smithi104:grafana.a> sudo journalctl -f -n 0 -u ceph-33168022-7814-11ef-baf6-efdc52797490@grafana.a.service 2024-09-21T12:28:40.602 INFO:tasks.cephadm:Setting up client nodes... 2024-09-21T12:28:40.602 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 33168022-7814-11ef-baf6-efdc52797490 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-09-21T12:28:40.795 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:28:40.916 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:40 smithi079 ceph-mon[30023]: Updating smithi079:/var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T12:28:40.916 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:40 smithi079 ceph-mon[30023]: Updating smithi104:/var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T12:28:40.916 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:40 smithi079 ceph-mon[30023]: Updating smithi079:/etc/ceph/ceph.client.admin.keyring 2024-09-21T12:28:40.916 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:40 smithi079 ceph-mon[30023]: Updating smithi104:/etc/ceph/ceph.client.admin.keyring 2024-09-21T12:28:40.916 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:40 smithi079 ceph-mon[30023]: from='client.24349 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi104=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:28:40.916 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:40 smithi079 ceph-mon[30023]: pgmap v3: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:28:40.916 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:40 smithi079 ceph-mon[30023]: Saving service grafana spec with placement smithi104=a;count:1 2024-09-21T12:28:40.916 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:40 smithi079 ceph-mon[30023]: Updating smithi079:/var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T12:28:40.916 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:40 smithi079 ceph-mon[30023]: Updating smithi104:/var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T12:28:40.916 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:40 smithi079 ceph-mon[30023]: [21/Sep/2024:12:28:39] ENGINE Bus STARTING 2024-09-21T12:28:40.917 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:40 smithi079 ceph-mon[30023]: [21/Sep/2024:12:28:39] ENGINE Serving on https://172.21.15.79:7151 2024-09-21T12:28:40.917 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:40 smithi079 ceph-mon[30023]: [21/Sep/2024:12:28:39] ENGINE Error in HTTPServer.serve 2024-09-21T12:28:40.917 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2024-09-21T12:28:40.917 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T12:28:40.917 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._connections.run(self.expiration_interval) 2024-09-21T12:28:40.917 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T12:28:40.917 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._run(expiration_interval) 2024-09-21T12:28:40.917 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T12:28:40.917 INFO:journalctl@ceph.mon.c.smithi079.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T12:28:40.917 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T12:28:40.917 INFO:journalctl@ceph.mon.c.smithi079.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T12:28:40.917 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T12:28:40.917 INFO:journalctl@ceph.mon.c.smithi079.stdout: s = self.context.wrap_socket( 2024-09-21T12:28:40.917 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T12:28:40.918 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.sslsocket_class._create( 2024-09-21T12:28:40.918 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T12:28:40.918 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.do_handshake() 2024-09-21T12:28:40.918 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T12:28:40.918 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._sslobj.do_handshake() 2024-09-21T12:28:40.918 INFO:journalctl@ceph.mon.c.smithi079.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T12:28:40.918 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:40 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:28:40.918 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:40 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:28:40.918 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:40 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:28:40.918 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:40 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:28:40.918 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:40 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:28:40.918 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:40 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:28:40.919 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:40 smithi079 ceph-mon[23699]: Updating smithi079:/var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T12:28:40.919 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:40 smithi079 ceph-mon[23699]: Updating smithi104:/var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T12:28:40.919 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:40 smithi079 ceph-mon[23699]: Updating smithi079:/etc/ceph/ceph.client.admin.keyring 2024-09-21T12:28:40.919 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:40 smithi079 ceph-mon[23699]: Updating smithi104:/etc/ceph/ceph.client.admin.keyring 2024-09-21T12:28:40.919 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:40 smithi079 ceph-mon[23699]: from='client.24349 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi104=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:28:40.919 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:40 smithi079 ceph-mon[23699]: pgmap v3: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:28:40.919 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:40 smithi079 ceph-mon[23699]: Saving service grafana spec with placement smithi104=a;count:1 2024-09-21T12:28:40.919 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:40 smithi079 ceph-mon[23699]: Updating smithi079:/var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T12:28:40.919 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:40 smithi079 ceph-mon[23699]: Updating smithi104:/var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T12:28:40.919 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:40 smithi079 ceph-mon[23699]: [21/Sep/2024:12:28:39] ENGINE Bus STARTING 2024-09-21T12:28:40.919 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:40 smithi079 ceph-mon[23699]: [21/Sep/2024:12:28:39] ENGINE Serving on https://172.21.15.79:7151 2024-09-21T12:28:40.919 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:40 smithi079 ceph-mon[23699]: [21/Sep/2024:12:28:39] ENGINE Error in HTTPServer.serve 2024-09-21T12:28:40.920 INFO:journalctl@ceph.mon.a.smithi079.stdout: Traceback (most recent call last): 2024-09-21T12:28:40.920 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T12:28:40.920 INFO:journalctl@ceph.mon.a.smithi079.stdout: self._connections.run(self.expiration_interval) 2024-09-21T12:28:40.920 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T12:28:40.920 INFO:journalctl@ceph.mon.a.smithi079.stdout: self._run(expiration_interval) 2024-09-21T12:28:40.920 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T12:28:40.920 INFO:journalctl@ceph.mon.a.smithi079.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T12:28:40.920 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T12:28:40.920 INFO:journalctl@ceph.mon.a.smithi079.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T12:28:40.920 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T12:28:40.920 INFO:journalctl@ceph.mon.a.smithi079.stdout: s = self.context.wrap_socket( 2024-09-21T12:28:40.920 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T12:28:40.920 INFO:journalctl@ceph.mon.a.smithi079.stdout: return self.sslsocket_class._create( 2024-09-21T12:28:40.920 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T12:28:40.920 INFO:journalctl@ceph.mon.a.smithi079.stdout: self.do_handshake() 2024-09-21T12:28:40.921 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T12:28:40.921 INFO:journalctl@ceph.mon.a.smithi079.stdout: self._sslobj.do_handshake() 2024-09-21T12:28:40.921 INFO:journalctl@ceph.mon.a.smithi079.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T12:28:40.921 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:40 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:28:40.921 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:40 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:28:40.921 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:40 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:28:40.921 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:40 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:28:40.921 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:40 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:28:40.921 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:40 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:28:40.926 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:40 smithi104 ceph-mon[25639]: Updating smithi079:/var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T12:28:40.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:40 smithi104 ceph-mon[25639]: Updating smithi104:/var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T12:28:40.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:40 smithi104 ceph-mon[25639]: Updating smithi079:/etc/ceph/ceph.client.admin.keyring 2024-09-21T12:28:40.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:40 smithi104 ceph-mon[25639]: Updating smithi104:/etc/ceph/ceph.client.admin.keyring 2024-09-21T12:28:40.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:40 smithi104 ceph-mon[25639]: from='client.24349 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi104=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:28:40.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:40 smithi104 ceph-mon[25639]: pgmap v3: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:28:40.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:40 smithi104 ceph-mon[25639]: Saving service grafana spec with placement smithi104=a;count:1 2024-09-21T12:28:40.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:40 smithi104 ceph-mon[25639]: Updating smithi079:/var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T12:28:40.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:40 smithi104 ceph-mon[25639]: Updating smithi104:/var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T12:28:40.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:40 smithi104 ceph-mon[25639]: [21/Sep/2024:12:28:39] ENGINE Bus STARTING 2024-09-21T12:28:40.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:40 smithi104 ceph-mon[25639]: [21/Sep/2024:12:28:39] ENGINE Serving on https://172.21.15.79:7151 2024-09-21T12:28:40.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:40 smithi104 ceph-mon[25639]: [21/Sep/2024:12:28:39] ENGINE Error in HTTPServer.serve 2024-09-21T12:28:40.927 INFO:journalctl@ceph.mon.b.smithi104.stdout: Traceback (most recent call last): 2024-09-21T12:28:40.927 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T12:28:40.928 INFO:journalctl@ceph.mon.b.smithi104.stdout: self._connections.run(self.expiration_interval) 2024-09-21T12:28:40.928 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T12:28:40.928 INFO:journalctl@ceph.mon.b.smithi104.stdout: self._run(expiration_interval) 2024-09-21T12:28:40.928 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T12:28:40.928 INFO:journalctl@ceph.mon.b.smithi104.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T12:28:40.928 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T12:28:40.928 INFO:journalctl@ceph.mon.b.smithi104.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T12:28:40.928 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T12:28:40.928 INFO:journalctl@ceph.mon.b.smithi104.stdout: s = self.context.wrap_socket( 2024-09-21T12:28:40.928 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T12:28:40.928 INFO:journalctl@ceph.mon.b.smithi104.stdout: return self.sslsocket_class._create( 2024-09-21T12:28:40.928 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T12:28:40.928 INFO:journalctl@ceph.mon.b.smithi104.stdout: self.do_handshake() 2024-09-21T12:28:40.928 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T12:28:40.928 INFO:journalctl@ceph.mon.b.smithi104.stdout: self._sslobj.do_handshake() 2024-09-21T12:28:40.929 INFO:journalctl@ceph.mon.b.smithi104.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T12:28:40.929 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:40 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:28:40.929 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:40 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:28:40.929 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:40 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:28:40.929 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:40 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:28:40.929 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:40 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:28:40.929 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:40 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:28:41.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:41 smithi104 ceph-mon[25639]: Deploying daemon alertmanager.a on smithi079 2024-09-21T12:28:41.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:41 smithi104 ceph-mon[25639]: [21/Sep/2024:12:28:39] ENGINE Serving on http://172.21.15.79:8765 2024-09-21T12:28:41.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:41 smithi104 ceph-mon[25639]: [21/Sep/2024:12:28:39] ENGINE Bus STARTED 2024-09-21T12:28:41.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:41 smithi104 ceph-mon[25639]: pgmap v4: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:28:41.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:41 smithi104 ceph-mon[25639]: mgrmap e19: y(active, since 2s), standbys: x 2024-09-21T12:28:41.941 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:41 smithi079 ceph-mon[30023]: Deploying daemon alertmanager.a on smithi079 2024-09-21T12:28:41.942 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:41 smithi079 ceph-mon[30023]: [21/Sep/2024:12:28:39] ENGINE Serving on http://172.21.15.79:8765 2024-09-21T12:28:41.942 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:41 smithi079 ceph-mon[30023]: [21/Sep/2024:12:28:39] ENGINE Bus STARTED 2024-09-21T12:28:41.942 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:41 smithi079 ceph-mon[30023]: pgmap v4: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:28:41.942 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:41 smithi079 ceph-mon[30023]: mgrmap e19: y(active, since 2s), standbys: x 2024-09-21T12:28:41.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:41 smithi079 ceph-mon[23699]: Deploying daemon alertmanager.a on smithi079 2024-09-21T12:28:41.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:41 smithi079 ceph-mon[23699]: [21/Sep/2024:12:28:39] ENGINE Serving on http://172.21.15.79:8765 2024-09-21T12:28:41.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:41 smithi079 ceph-mon[23699]: [21/Sep/2024:12:28:39] ENGINE Bus STARTED 2024-09-21T12:28:41.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:41 smithi079 ceph-mon[23699]: pgmap v4: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:28:41.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:41 smithi079 ceph-mon[23699]: mgrmap e19: y(active, since 2s), standbys: x 2024-09-21T12:28:43.230 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:28:43 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: ::ffff:172.21.15.104 - - [21/Sep/2024:12:28:43] "GET /metrics HTTP/1.1" 200 23677 "" "Prometheus/2.43.0" 2024-09-21T12:28:43.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:43 smithi079 ceph-mon[23699]: pgmap v5: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:28:43.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:43 smithi079 ceph-mon[23699]: mgrmap e20: y(active, since 4s), standbys: x 2024-09-21T12:28:43.942 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:43 smithi079 ceph-mon[30023]: pgmap v5: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:28:43.942 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:43 smithi079 ceph-mon[30023]: mgrmap e20: y(active, since 4s), standbys: x 2024-09-21T12:28:44.176 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:43 smithi104 ceph-mon[25639]: pgmap v5: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:28:44.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:43 smithi104 ceph-mon[25639]: mgrmap e20: y(active, since 4s), standbys: x 2024-09-21T12:28:44.812 INFO:teuthology.orchestra.run.smithi079.stdout:[client.0] 2024-09-21T12:28:44.812 INFO:teuthology.orchestra.run.smithi079.stdout: key = AQD8u+5m4o4fMBAAkMuBh3x9e090YWr2Q5S5GA== 2024-09-21T12:28:44.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:44 smithi079 ceph-mon[23699]: pgmap v6: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:28:44.942 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:44 smithi079 ceph-mon[30023]: pgmap v6: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:28:45.176 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:44 smithi104 ceph-mon[25639]: pgmap v6: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:28:46.046 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:45 smithi079 ceph-mon[23699]: from='client.? 172.21.15.79:0/2279764378' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-21T12:28:46.046 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:45 smithi079 ceph-mon[23699]: from='client.? 172.21.15.79:0/2279764378' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-21T12:28:46.046 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:45 smithi079 ceph-mon[30023]: from='client.? 172.21.15.79:0/2279764378' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-21T12:28:46.046 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:45 smithi079 ceph-mon[30023]: from='client.? 172.21.15.79:0/2279764378' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-21T12:28:46.070 DEBUG:teuthology.orchestra.run.smithi079:> set -ex 2024-09-21T12:28:46.070 DEBUG:teuthology.orchestra.run.smithi079:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-09-21T12:28:46.070 DEBUG:teuthology.orchestra.run.smithi079:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-21T12:28:46.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:45 smithi104 ceph-mon[25639]: from='client.? 172.21.15.79:0/2279764378' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-21T12:28:46.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:45 smithi104 ceph-mon[25639]: from='client.? 172.21.15.79:0/2279764378' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-21T12:28:46.112 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 33168022-7814-11ef-baf6-efdc52797490 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-09-21T12:28:47.073 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:46 smithi104 ceph-mon[25639]: pgmap v7: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-09-21T12:28:47.184 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:46 smithi079 ceph-mon[23699]: pgmap v7: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-09-21T12:28:47.184 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:46 smithi079 ceph-mon[30023]: pgmap v7: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-09-21T12:28:47.568 INFO:teuthology.orchestra.run.smithi104.stderr:Inferring config /var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T12:28:49.192 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Sep 21 12:28:49 smithi079 systemd[1]: Starting Ceph alertmanager.a for 33168022-7814-11ef-baf6-efdc52797490... 2024-09-21T12:28:49.514 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:49 smithi104 ceph-mon[25639]: pgmap v8: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-21T12:28:49.687 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:49 smithi079 ceph-mon[23699]: pgmap v8: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-21T12:28:49.688 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:49 smithi079 ceph-mon[30023]: pgmap v8: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-21T12:28:49.688 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Sep 21 12:28:49 smithi079 podman[49534]: 2024-09-21 12:28:49.236349497 +0000 UTC m=+0.108880951 volume create 774c12552acd6d77ce3e8704b30a94cf7924221113135ed620099ade892f43a2 2024-09-21T12:28:49.688 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Sep 21 12:28:49 smithi079 podman[49534]: 2024-09-21 12:28:49.144156451 +0000 UTC m=+0.016687911 image pull c8568f914cd25b2062c44e9f79f9c18da6e3b85fe0c47a12a2191c61426c2b19 quay.io/prometheus/alertmanager:v0.25.0 2024-09-21T12:28:49.688 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Sep 21 12:28:49 smithi079 podman[49534]: 2024-09-21 12:28:49.336542366 +0000 UTC m=+0.209073820 container create 88e00126f964a731b22d0d46d762d8510fddd307155e6af704a19763801b254f (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-33168022-7814-11ef-baf6-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-21T12:28:49.955 INFO:teuthology.orchestra.run.smithi104.stdout:[client.1] 2024-09-21T12:28:49.955 INFO:teuthology.orchestra.run.smithi104.stdout: key = AQABvO5mUm+nOBAAyi3/BbjG0BSYzw7KRE1g8A== 2024-09-21T12:28:50.153 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Sep 21 12:28:49 smithi079 podman[49534]: 2024-09-21 12:28:49.79973359 +0000 UTC m=+0.672265045 container init 88e00126f964a731b22d0d46d762d8510fddd307155e6af704a19763801b254f (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-33168022-7814-11ef-baf6-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-21T12:28:50.153 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Sep 21 12:28:49 smithi079 podman[49534]: 2024-09-21 12:28:49.802930325 +0000 UTC m=+0.675461778 container start 88e00126f964a731b22d0d46d762d8510fddd307155e6af704a19763801b254f (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-33168022-7814-11ef-baf6-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-21T12:28:50.153 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Sep 21 12:28:49 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-alertmanager-a[49549]: ts=2024-09-21T12:28:49.815Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-09-21T12:28:50.153 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Sep 21 12:28:49 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-alertmanager-a[49549]: ts=2024-09-21T12:28:49.815Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-09-21T12:28:50.153 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Sep 21 12:28:49 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-alertmanager-a[49549]: ts=2024-09-21T12:28:49.817Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.79 port=9094 2024-09-21T12:28:50.153 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Sep 21 12:28:49 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-alertmanager-a[49549]: ts=2024-09-21T12:28:49.818Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-09-21T12:28:50.153 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Sep 21 12:28:49 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-alertmanager-a[49549]: ts=2024-09-21T12:28:49.841Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-21T12:28:50.153 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Sep 21 12:28:49 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-alertmanager-a[49549]: ts=2024-09-21T12:28:49.842Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-21T12:28:50.153 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Sep 21 12:28:49 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-alertmanager-a[49549]: ts=2024-09-21T12:28:49.844Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-09-21T12:28:50.153 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Sep 21 12:28:49 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-alertmanager-a[49549]: ts=2024-09-21T12:28:49.844Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-09-21T12:28:50.153 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Sep 21 12:28:49 smithi079 bash[49534]: 88e00126f964a731b22d0d46d762d8510fddd307155e6af704a19763801b254f 2024-09-21T12:28:50.154 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Sep 21 12:28:49 smithi079 systemd[1]: Started Ceph alertmanager.a for 33168022-7814-11ef-baf6-efdc52797490. 2024-09-21T12:28:50.437 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:50 smithi079 ceph-mon[23699]: from='client.? 172.21.15.104:0/2976611699' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-21T12:28:50.438 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:50 smithi079 ceph-mon[23699]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-21T12:28:50.438 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:50 smithi079 ceph-mon[23699]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-21T12:28:50.438 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:50 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:28:50.438 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:50 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:28:50.438 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:50 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:28:50.438 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:50 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:28:50.438 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:50 smithi079 ceph-mon[30023]: from='client.? 172.21.15.104:0/2976611699' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-21T12:28:50.438 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:50 smithi079 ceph-mon[30023]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-21T12:28:50.438 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:50 smithi079 ceph-mon[30023]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-21T12:28:50.439 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:50 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:28:50.439 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:50 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:28:50.439 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:50 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:28:50.439 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:50 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:28:50.515 DEBUG:teuthology.orchestra.run.smithi104:> set -ex 2024-09-21T12:28:50.515 DEBUG:teuthology.orchestra.run.smithi104:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-09-21T12:28:50.515 DEBUG:teuthology.orchestra.run.smithi104:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-09-21T12:28:50.555 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 33168022-7814-11ef-baf6-efdc52797490 -- ceph config log 1 --format=json 2024-09-21T12:28:50.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:50 smithi104 ceph-mon[25639]: from='client.? 172.21.15.104:0/2976611699' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-21T12:28:50.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:50 smithi104 ceph-mon[25639]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-21T12:28:50.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:50 smithi104 ceph-mon[25639]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-21T12:28:50.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:50 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:28:50.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:50 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:28:50.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:50 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:28:50.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:50 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:28:50.758 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:28:51.663 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:51 smithi079 ceph-mon[23699]: Deploying daemon node-exporter.a on smithi079 2024-09-21T12:28:51.663 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:51 smithi079 ceph-mon[23699]: pgmap v9: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-21T12:28:51.663 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:51 smithi079 ceph-mon[30023]: Deploying daemon node-exporter.a on smithi079 2024-09-21T12:28:51.663 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:51 smithi079 ceph-mon[30023]: pgmap v9: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-21T12:28:51.676 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:51 smithi104 ceph-mon[25639]: Deploying daemon node-exporter.a on smithi079 2024-09-21T12:28:51.676 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:51 smithi104 ceph-mon[25639]: pgmap v9: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-21T12:28:51.942 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Sep 21 12:28:51 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-alertmanager-a[49549]: ts=2024-09-21T12:28:51.818Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000325631s 2024-09-21T12:28:53.395 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:28:53 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: ::ffff:172.21.15.104 - - [21/Sep/2024:12:28:53] "GET /metrics HTTP/1.1" 200 23677 "" "Prometheus/2.43.0" 2024-09-21T12:28:53.666 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:53 smithi079 ceph-mon[23699]: pgmap v10: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-21T12:28:53.666 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:53 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T12:28:53.666 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:53 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:28:53.667 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:53 smithi079 ceph-mon[30023]: pgmap v10: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-21T12:28:53.667 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:53 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T12:28:53.667 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:53 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:28:53.676 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:53 smithi104 ceph-mon[25639]: pgmap v10: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-21T12:28:53.676 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:53 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T12:28:53.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:53 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:28:53.942 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Sep 21 12:28:53 smithi079 systemd[1]: Starting Ceph node-exporter.a for 33168022-7814-11ef-baf6-efdc52797490... 2024-09-21T12:28:54.528 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-21T12:28:55.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:55 smithi079 ceph-mon[23699]: pgmap v11: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-21T12:28:55.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:55 smithi079 ceph-mon[23699]: from='client.? 172.21.15.79:0/3386382216' entity='client.admin' cmd=[{"prefix": "config log", "num": 1, "format": "json"}]: dispatch 2024-09-21T12:28:55.442 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:55 smithi079 ceph-mon[30023]: pgmap v11: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-21T12:28:55.442 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:55 smithi079 ceph-mon[30023]: from='client.? 172.21.15.79:0/3386382216' entity='client.admin' cmd=[{"prefix": "config log", "num": 1, "format": "json"}]: dispatch 2024-09-21T12:28:55.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:55 smithi104 ceph-mon[25639]: pgmap v11: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-21T12:28:55.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:55 smithi104 ceph-mon[25639]: from='client.? 172.21.15.79:0/3386382216' entity='client.admin' cmd=[{"prefix": "config log", "num": 1, "format": "json"}]: dispatch 2024-09-21T12:28:55.947 INFO:teuthology.orchestra.run.smithi079.stdout:[{"version":24,"timestamp":"2024-09-21T12:28:38.367906+0000","name":"","changes":[{"name":"osd/host:smithi079/osd_memory_target","previous_value":"3942673817"}]}] 2024-09-21T12:28:55.948 INFO:tasks.ceph_manager:config epoch is 24 2024-09-21T12:28:55.948 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-21T12:28:55.948 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-09-21T12:28:55.948 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 33168022-7814-11ef-baf6-efdc52797490 -- ceph mgr dump --format=json 2024-09-21T12:28:56.168 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:28:56.442 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Sep 21 12:28:56 smithi079 bash[50262]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-09-21T12:28:56.942 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Sep 21 12:28:56 smithi079 bash[50262]: Getting image source signatures 2024-09-21T12:28:56.942 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Sep 21 12:28:56 smithi079 bash[50262]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-09-21T12:28:56.942 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Sep 21 12:28:56 smithi079 bash[50262]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-09-21T12:28:56.942 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Sep 21 12:28:56 smithi079 bash[50262]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-09-21T12:28:57.676 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:57 smithi104 ceph-mon[25639]: pgmap v12: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-21T12:28:57.691 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:57 smithi079 ceph-mon[30023]: pgmap v12: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-21T12:28:57.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:57 smithi079 ceph-mon[23699]: pgmap v12: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-21T12:28:58.510 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Sep 21 12:28:58 smithi079 bash[50262]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-09-21T12:28:58.746 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-21T12:28:58.942 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Sep 21 12:28:58 smithi079 bash[50262]: Writing manifest to image destination 2024-09-21T12:28:59.270 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Sep 21 12:28:58 smithi079 podman[50262]: 2024-09-21 12:28:58.882473556 +0000 UTC m=+2.827772427 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-09-21T12:28:59.270 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Sep 21 12:28:59 smithi079 podman[50262]: 2024-09-21 12:28:59.15670069 +0000 UTC m=+3.101999562 container create f7bef67502dce5dc1f830f29a2e1911744e2a54a91efc7b6a8bb02a9e03c7f9e (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-33168022-7814-11ef-baf6-efdc52797490-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-21T12:28:59.676 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:59 smithi104 ceph-mon[25639]: pgmap v13: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:28:59.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:28:59 smithi104 ceph-mon[25639]: from='client.? 172.21.15.79:0/3123996919' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-09-21T12:28:59.691 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:59 smithi079 ceph-mon[23699]: pgmap v13: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:28:59.691 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:28:59 smithi079 ceph-mon[23699]: from='client.? 172.21.15.79:0/3123996919' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-09-21T12:28:59.692 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:59 smithi079 ceph-mon[30023]: pgmap v13: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:28:59.692 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:28:59 smithi079 ceph-mon[30023]: from='client.? 172.21.15.79:0/3123996919' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-09-21T12:28:59.692 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Sep 21 12:28:59 smithi079 podman[50262]: 2024-09-21 12:28:59.37543112 +0000 UTC m=+3.320729992 container init f7bef67502dce5dc1f830f29a2e1911744e2a54a91efc7b6a8bb02a9e03c7f9e (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-33168022-7814-11ef-baf6-efdc52797490-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-21T12:28:59.692 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Sep 21 12:28:59 smithi079 podman[50262]: 2024-09-21 12:28:59.378342431 +0000 UTC m=+3.323641312 container start f7bef67502dce5dc1f830f29a2e1911744e2a54a91efc7b6a8bb02a9e03c7f9e (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-33168022-7814-11ef-baf6-efdc52797490-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-21T12:28:59.693 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Sep 21 12:28:59 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-node-exporter-a[50573]: ts=2024-09-21T12:28:59.382Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-09-21T12:28:59.693 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Sep 21 12:28:59 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-node-exporter-a[50573]: ts=2024-09-21T12:28:59.382Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-09-21T12:28:59.693 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Sep 21 12:28:59 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-node-exporter-a[50573]: ts=2024-09-21T12:28:59.383Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-09-21T12:28:59.693 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Sep 21 12:28:59 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-node-exporter-a[50573]: ts=2024-09-21T12:28:59.383Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-09-21T12:28:59.693 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Sep 21 12:28:59 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-node-exporter-a[50573]: ts=2024-09-21T12:28:59.383Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-09-21T12:28:59.693 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Sep 21 12:28:59 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-node-exporter-a[50573]: ts=2024-09-21T12:28:59.383Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-09-21T12:28:59.693 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Sep 21 12:28:59 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-node-exporter-a[50573]: ts=2024-09-21T12:28:59.383Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-09-21T12:28:59.693 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Sep 21 12:28:59 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-node-exporter-a[50573]: ts=2024-09-21T12:28:59.383Z caller=node_exporter.go:117 level=info collector=arp 2024-09-21T12:28:59.693 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Sep 21 12:28:59 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-node-exporter-a[50573]: ts=2024-09-21T12:28:59.383Z caller=node_exporter.go:117 level=info collector=bcache 2024-09-21T12:28:59.693 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Sep 21 12:28:59 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-node-exporter-a[50573]: ts=2024-09-21T12:28:59.383Z caller=node_exporter.go:117 level=info collector=bonding 2024-09-21T12:28:59.693 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Sep 21 12:28:59 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-node-exporter-a[50573]: ts=2024-09-21T12:28:59.383Z caller=node_exporter.go:117 level=info collector=btrfs 2024-09-21T12:28:59.693 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Sep 21 12:28:59 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-node-exporter-a[50573]: ts=2024-09-21T12:28:59.383Z caller=node_exporter.go:117 level=info collector=conntrack 2024-09-21T12:28:59.693 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Sep 21 12:28:59 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-node-exporter-a[50573]: ts=2024-09-21T12:28:59.383Z caller=node_exporter.go:117 level=info collector=cpu 2024-09-21T12:28:59.694 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Sep 21 12:28:59 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-node-exporter-a[50573]: ts=2024-09-21T12:28:59.383Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-09-21T12:28:59.694 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Sep 21 12:28:59 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-node-exporter-a[50573]: ts=2024-09-21T12:28:59.383Z caller=node_exporter.go:117 level=info collector=diskstats 2024-09-21T12:28:59.694 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Sep 21 12:28:59 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-node-exporter-a[50573]: ts=2024-09-21T12:28:59.383Z caller=node_exporter.go:117 level=info collector=dmi 2024-09-21T12:28:59.694 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Sep 21 12:28:59 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-node-exporter-a[50573]: ts=2024-09-21T12:28:59.383Z caller=node_exporter.go:117 level=info collector=edac 2024-09-21T12:28:59.694 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Sep 21 12:28:59 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-node-exporter-a[50573]: ts=2024-09-21T12:28:59.383Z caller=node_exporter.go:117 level=info collector=entropy 2024-09-21T12:28:59.695 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Sep 21 12:28:59 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-node-exporter-a[50573]: ts=2024-09-21T12:28:59.383Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-09-21T12:28:59.695 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Sep 21 12:28:59 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-node-exporter-a[50573]: ts=2024-09-21T12:28:59.383Z caller=node_exporter.go:117 level=info collector=filefd 2024-09-21T12:28:59.695 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Sep 21 12:28:59 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-node-exporter-a[50573]: ts=2024-09-21T12:28:59.383Z caller=node_exporter.go:117 level=info collector=filesystem 2024-09-21T12:28:59.695 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Sep 21 12:28:59 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-node-exporter-a[50573]: ts=2024-09-21T12:28:59.383Z caller=node_exporter.go:117 level=info collector=hwmon 2024-09-21T12:28:59.695 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Sep 21 12:28:59 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-node-exporter-a[50573]: ts=2024-09-21T12:28:59.383Z caller=node_exporter.go:117 level=info collector=infiniband 2024-09-21T12:28:59.695 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Sep 21 12:28:59 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-node-exporter-a[50573]: ts=2024-09-21T12:28:59.383Z caller=node_exporter.go:117 level=info collector=ipvs 2024-09-21T12:28:59.695 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Sep 21 12:28:59 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-node-exporter-a[50573]: ts=2024-09-21T12:28:59.383Z caller=node_exporter.go:117 level=info collector=loadavg 2024-09-21T12:28:59.696 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Sep 21 12:28:59 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-node-exporter-a[50573]: ts=2024-09-21T12:28:59.383Z caller=node_exporter.go:117 level=info collector=mdadm 2024-09-21T12:28:59.696 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Sep 21 12:28:59 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-node-exporter-a[50573]: ts=2024-09-21T12:28:59.383Z caller=node_exporter.go:117 level=info collector=meminfo 2024-09-21T12:28:59.696 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Sep 21 12:28:59 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-node-exporter-a[50573]: ts=2024-09-21T12:28:59.383Z caller=node_exporter.go:117 level=info collector=netclass 2024-09-21T12:28:59.696 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Sep 21 12:28:59 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-node-exporter-a[50573]: ts=2024-09-21T12:28:59.383Z caller=node_exporter.go:117 level=info collector=netdev 2024-09-21T12:28:59.696 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Sep 21 12:28:59 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-node-exporter-a[50573]: ts=2024-09-21T12:28:59.383Z caller=node_exporter.go:117 level=info collector=netstat 2024-09-21T12:28:59.696 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Sep 21 12:28:59 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-node-exporter-a[50573]: ts=2024-09-21T12:28:59.383Z caller=node_exporter.go:117 level=info collector=nfs 2024-09-21T12:28:59.696 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Sep 21 12:28:59 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-node-exporter-a[50573]: ts=2024-09-21T12:28:59.383Z caller=node_exporter.go:117 level=info collector=nfsd 2024-09-21T12:28:59.696 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Sep 21 12:28:59 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-node-exporter-a[50573]: ts=2024-09-21T12:28:59.383Z caller=node_exporter.go:117 level=info collector=nvme 2024-09-21T12:28:59.697 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Sep 21 12:28:59 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-node-exporter-a[50573]: ts=2024-09-21T12:28:59.383Z caller=node_exporter.go:117 level=info collector=os 2024-09-21T12:28:59.697 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Sep 21 12:28:59 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-node-exporter-a[50573]: ts=2024-09-21T12:28:59.383Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-09-21T12:28:59.697 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Sep 21 12:28:59 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-node-exporter-a[50573]: ts=2024-09-21T12:28:59.383Z caller=node_exporter.go:117 level=info collector=pressure 2024-09-21T12:28:59.697 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Sep 21 12:28:59 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-node-exporter-a[50573]: ts=2024-09-21T12:28:59.383Z caller=node_exporter.go:117 level=info collector=rapl 2024-09-21T12:28:59.697 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Sep 21 12:28:59 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-node-exporter-a[50573]: ts=2024-09-21T12:28:59.383Z caller=node_exporter.go:117 level=info collector=schedstat 2024-09-21T12:28:59.697 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Sep 21 12:28:59 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-node-exporter-a[50573]: ts=2024-09-21T12:28:59.383Z caller=node_exporter.go:117 level=info collector=selinux 2024-09-21T12:28:59.697 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Sep 21 12:28:59 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-node-exporter-a[50573]: ts=2024-09-21T12:28:59.383Z caller=node_exporter.go:117 level=info collector=sockstat 2024-09-21T12:28:59.698 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Sep 21 12:28:59 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-node-exporter-a[50573]: ts=2024-09-21T12:28:59.383Z caller=node_exporter.go:117 level=info collector=softnet 2024-09-21T12:28:59.698 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Sep 21 12:28:59 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-node-exporter-a[50573]: ts=2024-09-21T12:28:59.383Z caller=node_exporter.go:117 level=info collector=stat 2024-09-21T12:28:59.698 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Sep 21 12:28:59 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-node-exporter-a[50573]: ts=2024-09-21T12:28:59.383Z caller=node_exporter.go:117 level=info collector=tapestats 2024-09-21T12:28:59.698 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Sep 21 12:28:59 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-node-exporter-a[50573]: ts=2024-09-21T12:28:59.383Z caller=node_exporter.go:117 level=info collector=textfile 2024-09-21T12:28:59.698 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Sep 21 12:28:59 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-node-exporter-a[50573]: ts=2024-09-21T12:28:59.383Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-09-21T12:28:59.698 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Sep 21 12:28:59 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-node-exporter-a[50573]: ts=2024-09-21T12:28:59.383Z caller=node_exporter.go:117 level=info collector=time 2024-09-21T12:28:59.698 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Sep 21 12:28:59 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-node-exporter-a[50573]: ts=2024-09-21T12:28:59.383Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-09-21T12:28:59.698 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Sep 21 12:28:59 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-node-exporter-a[50573]: ts=2024-09-21T12:28:59.383Z caller=node_exporter.go:117 level=info collector=uname 2024-09-21T12:28:59.699 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Sep 21 12:28:59 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-node-exporter-a[50573]: ts=2024-09-21T12:28:59.383Z caller=node_exporter.go:117 level=info collector=vmstat 2024-09-21T12:28:59.699 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Sep 21 12:28:59 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-node-exporter-a[50573]: ts=2024-09-21T12:28:59.383Z caller=node_exporter.go:117 level=info collector=xfs 2024-09-21T12:28:59.699 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Sep 21 12:28:59 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-node-exporter-a[50573]: ts=2024-09-21T12:28:59.383Z caller=node_exporter.go:117 level=info collector=zfs 2024-09-21T12:28:59.699 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Sep 21 12:28:59 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-node-exporter-a[50573]: ts=2024-09-21T12:28:59.384Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-09-21T12:28:59.699 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Sep 21 12:28:59 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-node-exporter-a[50573]: ts=2024-09-21T12:28:59.384Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-09-21T12:28:59.699 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Sep 21 12:28:59 smithi079 bash[50262]: f7bef67502dce5dc1f830f29a2e1911744e2a54a91efc7b6a8bb02a9e03c7f9e 2024-09-21T12:28:59.699 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Sep 21 12:28:59 smithi079 systemd[1]: Started Ceph node-exporter.a for 33168022-7814-11ef-baf6-efdc52797490. 2024-09-21T12:28:59.993 INFO:teuthology.orchestra.run.smithi079.stdout:{"epoch":20,"flags":0,"active_gid":24341,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6800","nonce":2765998793},{"type":"v1","addr":"172.21.15.79:6801","nonce":2765998793}]},"active_addr":"172.21.15.79:6801/2765998793","active_change":"2024-09-21T12:28:38.322263+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":24340,"name":"x","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.79:8443/","prometheus":"http://172.21.15.79:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":59,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.79:0","nonce":1760224796}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.79:0","nonce":3835416297}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.79:0","nonce":2336253920}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.79:0","nonce":3698254455}]}]} 2024-09-21T12:28:59.996 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-09-21T12:28:59.996 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-09-21T12:28:59.996 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 33168022-7814-11ef-baf6-efdc52797490 -- ceph osd dump --format=json 2024-09-21T12:29:00.121 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Sep 21 12:28:59 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-alertmanager-a[49549]: ts=2024-09-21T12:28:59.820Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.002570066s 2024-09-21T12:29:00.423 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 12:29:00 smithi104 systemd[1]: Starting Ceph node-exporter.b for 33168022-7814-11ef-baf6-efdc52797490... 2024-09-21T12:29:00.677 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 12:29:00 smithi104 bash[46556]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-09-21T12:29:00.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:29:00 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:00.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:29:00 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:00.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:29:00 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:00.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:29:00 smithi104 ceph-mon[25639]: Deploying daemon node-exporter.b on smithi104 2024-09-21T12:29:00.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:29:00 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:00.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:29:00 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:00.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:29:00 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:00.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:29:00 smithi079 ceph-mon[23699]: Deploying daemon node-exporter.b on smithi104 2024-09-21T12:29:00.692 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:29:00 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:00.692 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:29:00 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:00.692 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:29:00 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:00.692 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:29:00 smithi079 ceph-mon[30023]: Deploying daemon node-exporter.b on smithi104 2024-09-21T12:29:01.147 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 12:29:00 smithi104 bash[46556]: Getting image source signatures 2024-09-21T12:29:01.148 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 12:29:00 smithi104 bash[46556]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-09-21T12:29:01.148 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 12:29:00 smithi104 bash[46556]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-09-21T12:29:01.148 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 12:29:00 smithi104 bash[46556]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-09-21T12:29:01.271 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:29:01.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:29:01 smithi079 ceph-mon[23699]: pgmap v14: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:29:01.692 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:29:01 smithi079 ceph-mon[30023]: pgmap v14: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:29:01.926 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:29:01 smithi104 ceph-mon[25639]: pgmap v14: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:29:03.364 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:29:03 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: ::ffff:172.21.15.104 - - [21/Sep/2024:12:29:03] "GET /metrics HTTP/1.1" 200 26201 "" "Prometheus/2.43.0" 2024-09-21T12:29:03.427 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 12:29:02 smithi104 bash[46556]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-09-21T12:29:03.427 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 12:29:03 smithi104 bash[46556]: Writing manifest to image destination 2024-09-21T12:29:03.620 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:29:03 smithi079 ceph-mon[23699]: pgmap v15: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:29:03.620 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:29:03 smithi079 ceph-mon[30023]: pgmap v15: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:29:03.727 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:29:03 smithi104 ceph-mon[25639]: pgmap v15: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:29:03.727 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 12:29:03 smithi104 podman[46556]: 2024-09-21 12:29:03.457603326 +0000 UTC m=+3.126469949 container create 3701f535d07838f1eab9dfdb3b98ec612ce03736f752b85af28317d7e811aac5 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-33168022-7814-11ef-baf6-efdc52797490-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-21T12:29:03.727 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 12:29:03 smithi104 podman[46556]: 2024-09-21 12:29:03.384344381 +0000 UTC m=+3.053211005 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-09-21T12:29:03.728 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 12:29:03 smithi104 podman[46556]: 2024-09-21 12:29:03.643097927 +0000 UTC m=+3.311964551 container init 3701f535d07838f1eab9dfdb3b98ec612ce03736f752b85af28317d7e811aac5 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-33168022-7814-11ef-baf6-efdc52797490-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-21T12:29:03.728 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 12:29:03 smithi104 podman[46556]: 2024-09-21 12:29:03.646181029 +0000 UTC m=+3.315047662 container start 3701f535d07838f1eab9dfdb3b98ec612ce03736f752b85af28317d7e811aac5 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-33168022-7814-11ef-baf6-efdc52797490-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-21T12:29:03.728 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 12:29:03 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-node-exporter-b[46664]: ts=2024-09-21T12:29:03.649Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-09-21T12:29:03.728 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 12:29:03 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-node-exporter-b[46664]: ts=2024-09-21T12:29:03.649Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-09-21T12:29:03.728 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 12:29:03 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-node-exporter-b[46664]: ts=2024-09-21T12:29:03.650Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-09-21T12:29:03.728 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 12:29:03 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-node-exporter-b[46664]: ts=2024-09-21T12:29:03.650Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-09-21T12:29:03.728 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 12:29:03 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-node-exporter-b[46664]: ts=2024-09-21T12:29:03.650Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-09-21T12:29:03.728 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 12:29:03 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-node-exporter-b[46664]: ts=2024-09-21T12:29:03.650Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-09-21T12:29:03.728 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 12:29:03 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-node-exporter-b[46664]: ts=2024-09-21T12:29:03.650Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-09-21T12:29:03.728 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 12:29:03 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-node-exporter-b[46664]: ts=2024-09-21T12:29:03.650Z caller=node_exporter.go:117 level=info collector=arp 2024-09-21T12:29:03.728 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 12:29:03 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-node-exporter-b[46664]: ts=2024-09-21T12:29:03.650Z caller=node_exporter.go:117 level=info collector=bcache 2024-09-21T12:29:03.728 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 12:29:03 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-node-exporter-b[46664]: ts=2024-09-21T12:29:03.650Z caller=node_exporter.go:117 level=info collector=bonding 2024-09-21T12:29:03.728 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 12:29:03 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-node-exporter-b[46664]: ts=2024-09-21T12:29:03.650Z caller=node_exporter.go:117 level=info collector=btrfs 2024-09-21T12:29:03.729 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 12:29:03 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-node-exporter-b[46664]: ts=2024-09-21T12:29:03.651Z caller=node_exporter.go:117 level=info collector=conntrack 2024-09-21T12:29:03.729 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 12:29:03 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-node-exporter-b[46664]: ts=2024-09-21T12:29:03.651Z caller=node_exporter.go:117 level=info collector=cpu 2024-09-21T12:29:03.729 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 12:29:03 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-node-exporter-b[46664]: ts=2024-09-21T12:29:03.651Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-09-21T12:29:03.729 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 12:29:03 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-node-exporter-b[46664]: ts=2024-09-21T12:29:03.651Z caller=node_exporter.go:117 level=info collector=diskstats 2024-09-21T12:29:03.729 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 12:29:03 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-node-exporter-b[46664]: ts=2024-09-21T12:29:03.651Z caller=node_exporter.go:117 level=info collector=dmi 2024-09-21T12:29:03.729 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 12:29:03 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-node-exporter-b[46664]: ts=2024-09-21T12:29:03.651Z caller=node_exporter.go:117 level=info collector=edac 2024-09-21T12:29:03.729 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 12:29:03 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-node-exporter-b[46664]: ts=2024-09-21T12:29:03.651Z caller=node_exporter.go:117 level=info collector=entropy 2024-09-21T12:29:03.729 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 12:29:03 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-node-exporter-b[46664]: ts=2024-09-21T12:29:03.651Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-09-21T12:29:03.729 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 12:29:03 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-node-exporter-b[46664]: ts=2024-09-21T12:29:03.651Z caller=node_exporter.go:117 level=info collector=filefd 2024-09-21T12:29:03.730 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 12:29:03 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-node-exporter-b[46664]: ts=2024-09-21T12:29:03.651Z caller=node_exporter.go:117 level=info collector=filesystem 2024-09-21T12:29:03.730 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 12:29:03 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-node-exporter-b[46664]: ts=2024-09-21T12:29:03.651Z caller=node_exporter.go:117 level=info collector=hwmon 2024-09-21T12:29:03.730 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 12:29:03 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-node-exporter-b[46664]: ts=2024-09-21T12:29:03.651Z caller=node_exporter.go:117 level=info collector=infiniband 2024-09-21T12:29:03.730 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 12:29:03 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-node-exporter-b[46664]: ts=2024-09-21T12:29:03.651Z caller=node_exporter.go:117 level=info collector=ipvs 2024-09-21T12:29:03.730 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 12:29:03 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-node-exporter-b[46664]: ts=2024-09-21T12:29:03.651Z caller=node_exporter.go:117 level=info collector=loadavg 2024-09-21T12:29:03.730 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 12:29:03 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-node-exporter-b[46664]: ts=2024-09-21T12:29:03.651Z caller=node_exporter.go:117 level=info collector=mdadm 2024-09-21T12:29:03.730 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 12:29:03 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-node-exporter-b[46664]: ts=2024-09-21T12:29:03.651Z caller=node_exporter.go:117 level=info collector=meminfo 2024-09-21T12:29:03.730 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 12:29:03 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-node-exporter-b[46664]: ts=2024-09-21T12:29:03.651Z caller=node_exporter.go:117 level=info collector=netclass 2024-09-21T12:29:03.730 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 12:29:03 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-node-exporter-b[46664]: ts=2024-09-21T12:29:03.651Z caller=node_exporter.go:117 level=info collector=netdev 2024-09-21T12:29:03.730 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 12:29:03 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-node-exporter-b[46664]: ts=2024-09-21T12:29:03.651Z caller=node_exporter.go:117 level=info collector=netstat 2024-09-21T12:29:03.730 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 12:29:03 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-node-exporter-b[46664]: ts=2024-09-21T12:29:03.651Z caller=node_exporter.go:117 level=info collector=nfs 2024-09-21T12:29:03.730 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 12:29:03 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-node-exporter-b[46664]: ts=2024-09-21T12:29:03.651Z caller=node_exporter.go:117 level=info collector=nfsd 2024-09-21T12:29:03.730 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 12:29:03 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-node-exporter-b[46664]: ts=2024-09-21T12:29:03.651Z caller=node_exporter.go:117 level=info collector=nvme 2024-09-21T12:29:03.731 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 12:29:03 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-node-exporter-b[46664]: ts=2024-09-21T12:29:03.651Z caller=node_exporter.go:117 level=info collector=os 2024-09-21T12:29:03.731 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 12:29:03 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-node-exporter-b[46664]: ts=2024-09-21T12:29:03.651Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-09-21T12:29:03.731 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 12:29:03 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-node-exporter-b[46664]: ts=2024-09-21T12:29:03.651Z caller=node_exporter.go:117 level=info collector=pressure 2024-09-21T12:29:03.731 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 12:29:03 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-node-exporter-b[46664]: ts=2024-09-21T12:29:03.651Z caller=node_exporter.go:117 level=info collector=rapl 2024-09-21T12:29:03.731 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 12:29:03 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-node-exporter-b[46664]: ts=2024-09-21T12:29:03.651Z caller=node_exporter.go:117 level=info collector=schedstat 2024-09-21T12:29:03.731 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 12:29:03 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-node-exporter-b[46664]: ts=2024-09-21T12:29:03.651Z caller=node_exporter.go:117 level=info collector=selinux 2024-09-21T12:29:03.731 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 12:29:03 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-node-exporter-b[46664]: ts=2024-09-21T12:29:03.651Z caller=node_exporter.go:117 level=info collector=sockstat 2024-09-21T12:29:03.731 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 12:29:03 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-node-exporter-b[46664]: ts=2024-09-21T12:29:03.651Z caller=node_exporter.go:117 level=info collector=softnet 2024-09-21T12:29:03.731 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 12:29:03 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-node-exporter-b[46664]: ts=2024-09-21T12:29:03.651Z caller=node_exporter.go:117 level=info collector=stat 2024-09-21T12:29:03.731 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 12:29:03 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-node-exporter-b[46664]: ts=2024-09-21T12:29:03.651Z caller=node_exporter.go:117 level=info collector=tapestats 2024-09-21T12:29:03.731 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 12:29:03 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-node-exporter-b[46664]: ts=2024-09-21T12:29:03.651Z caller=node_exporter.go:117 level=info collector=textfile 2024-09-21T12:29:03.731 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 12:29:03 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-node-exporter-b[46664]: ts=2024-09-21T12:29:03.651Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-09-21T12:29:03.731 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 12:29:03 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-node-exporter-b[46664]: ts=2024-09-21T12:29:03.651Z caller=node_exporter.go:117 level=info collector=time 2024-09-21T12:29:03.731 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 12:29:03 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-node-exporter-b[46664]: ts=2024-09-21T12:29:03.651Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-09-21T12:29:03.732 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 12:29:03 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-node-exporter-b[46664]: ts=2024-09-21T12:29:03.651Z caller=node_exporter.go:117 level=info collector=uname 2024-09-21T12:29:03.732 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 12:29:03 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-node-exporter-b[46664]: ts=2024-09-21T12:29:03.651Z caller=node_exporter.go:117 level=info collector=vmstat 2024-09-21T12:29:03.732 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 12:29:03 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-node-exporter-b[46664]: ts=2024-09-21T12:29:03.651Z caller=node_exporter.go:117 level=info collector=xfs 2024-09-21T12:29:03.732 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 12:29:03 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-node-exporter-b[46664]: ts=2024-09-21T12:29:03.651Z caller=node_exporter.go:117 level=info collector=zfs 2024-09-21T12:29:03.732 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 12:29:03 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-node-exporter-b[46664]: ts=2024-09-21T12:29:03.651Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-09-21T12:29:03.732 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 12:29:03 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-node-exporter-b[46664]: ts=2024-09-21T12:29:03.651Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-09-21T12:29:03.732 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 12:29:03 smithi104 bash[46556]: 3701f535d07838f1eab9dfdb3b98ec612ce03736f752b85af28317d7e811aac5 2024-09-21T12:29:04.055 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 12:29:03 smithi104 systemd[1]: Started Ceph node-exporter.b for 33168022-7814-11ef-baf6-efdc52797490. 2024-09-21T12:29:04.192 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:29:03 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: [21/Sep/2024:12:29:03] ENGINE Bus STOPPING 2024-09-21T12:29:04.192 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:29:04 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: [21/Sep/2024:12:29:04] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-21T12:29:04.192 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:29:04 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: [21/Sep/2024:12:29:04] ENGINE Bus STOPPED 2024-09-21T12:29:04.192 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:29:04 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: [21/Sep/2024:12:29:04] ENGINE Bus STARTING 2024-09-21T12:29:04.317 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-21T12:29:04.317 INFO:teuthology.orchestra.run.smithi079.stdout:{"epoch":59,"fsid":"33168022-7814-11ef-baf6-efdc52797490","created":"2024-09-21T12:23:21.896565+0000","modified":"2024-09-21T12:28:38.322154+0000","last_up_change":"2024-09-21T12:28:09.822191+0000","last_in_change":"2024-09-21T12:27:52.435732+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T12:26:19.078594+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"aa2c4408-1b95-4d23-9f60-d73b204290db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6802","nonce":846480130},{"type":"v1","addr":"172.21.15.79:6803","nonce":846480130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6804","nonce":846480130},{"type":"v1","addr":"172.21.15.79:6805","nonce":846480130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6808","nonce":846480130},{"type":"v1","addr":"172.21.15.79:6809","nonce":846480130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6806","nonce":846480130},{"type":"v1","addr":"172.21.15.79:6807","nonce":846480130}]},"public_addr":"172.21.15.79:6803/846480130","cluster_addr":"172.21.15.79:6805/846480130","heartbeat_back_addr":"172.21.15.79:6809/846480130","heartbeat_front_addr":"172.21.15.79:6807/846480130","state":["exists","up"]},{"osd":1,"uuid":"a05b4be6-79b2-4dde-bd51-14cafd833c5a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6810","nonce":1764722078},{"type":"v1","addr":"172.21.15.79:6811","nonce":1764722078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6812","nonce":1764722078},{"type":"v1","addr":"172.21.15.79:6813","nonce":1764722078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6816","nonce":1764722078},{"type":"v1","addr":"172.21.15.79:6817","nonce":1764722078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6814","nonce":1764722078},{"type":"v1","addr":"172.21.15.79:6815","nonce":1764722078}]},"public_addr":"172.21.15.79:6811/1764722078","cluster_addr":"172.21.15.79:6813/1764722078","heartbeat_back_addr":"172.21.15.79:6817/1764722078","heartbeat_front_addr":"172.21.15.79:6815/1764722078","state":["exists","up"]},{"osd":2,"uuid":"8b9ddbf7-8264-4e90-bcf0-e63a7f5e7e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6818","nonce":2982011382},{"type":"v1","addr":"172.21.15.79:6819","nonce":2982011382}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6820","nonce":2982011382},{"type":"v1","addr":"172.21.15.79:6821","nonce":2982011382}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6824","nonce":2982011382},{"type":"v1","addr":"172.21.15.79:6825","nonce":2982011382}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6822","nonce":2982011382},{"type":"v1","addr":"172.21.15.79:6823","nonce":2982011382}]},"public_addr":"172.21.15.79:6819/2982011382","cluster_addr":"172.21.15.79:6821/2982011382","heartbeat_back_addr":"172.21.15.79:6825/2982011382","heartbeat_front_addr":"172.21.15.79:6823/2982011382","state":["exists","up"]},{"osd":3,"uuid":"65ac97f0-572a-400f-977b-4c20cdc0fc9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6826","nonce":2894016442},{"type":"v1","addr":"172.21.15.79:6827","nonce":2894016442}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6828","nonce":2894016442},{"type":"v1","addr":"172.21.15.79:6829","nonce":2894016442}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6832","nonce":2894016442},{"type":"v1","addr":"172.21.15.79:6833","nonce":2894016442}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6830","nonce":2894016442},{"type":"v1","addr":"172.21.15.79:6831","nonce":2894016442}]},"public_addr":"172.21.15.79:6827/2894016442","cluster_addr":"172.21.15.79:6829/2894016442","heartbeat_back_addr":"172.21.15.79:6833/2894016442","heartbeat_front_addr":"172.21.15.79:6831/2894016442","state":["exists","up"]},{"osd":4,"uuid":"278156d6-8fd6-4852-9de2-1e11ea7e8917","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":3796154758},{"type":"v1","addr":"172.21.15.104:6801","nonce":3796154758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":3796154758},{"type":"v1","addr":"172.21.15.104:6803","nonce":3796154758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":3796154758},{"type":"v1","addr":"172.21.15.104:6807","nonce":3796154758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":3796154758},{"type":"v1","addr":"172.21.15.104:6805","nonce":3796154758}]},"public_addr":"172.21.15.104:6801/3796154758","cluster_addr":"172.21.15.104:6803/3796154758","heartbeat_back_addr":"172.21.15.104:6807/3796154758","heartbeat_front_addr":"172.21.15.104:6805/3796154758","state":["exists","up"]},{"osd":5,"uuid":"5435eabf-f4d1-4c86-b334-fa3acc9162c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":189336043},{"type":"v1","addr":"172.21.15.104:6809","nonce":189336043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":189336043},{"type":"v1","addr":"172.21.15.104:6811","nonce":189336043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":189336043},{"type":"v1","addr":"172.21.15.104:6815","nonce":189336043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":189336043},{"type":"v1","addr":"172.21.15.104:6813","nonce":189336043}]},"public_addr":"172.21.15.104:6809/189336043","cluster_addr":"172.21.15.104:6811/189336043","heartbeat_back_addr":"172.21.15.104:6815/189336043","heartbeat_front_addr":"172.21.15.104:6813/189336043","state":["exists","up"]},{"osd":6,"uuid":"df36e40d-a08e-4b0f-8774-4862209d0ba0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2903175260},{"type":"v1","addr":"172.21.15.104:6817","nonce":2903175260}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2903175260},{"type":"v1","addr":"172.21.15.104:6819","nonce":2903175260}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2903175260},{"type":"v1","addr":"172.21.15.104:6823","nonce":2903175260}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2903175260},{"type":"v1","addr":"172.21.15.104:6821","nonce":2903175260}]},"public_addr":"172.21.15.104:6817/2903175260","cluster_addr":"172.21.15.104:6819/2903175260","heartbeat_back_addr":"172.21.15.104:6823/2903175260","heartbeat_front_addr":"172.21.15.104:6821/2903175260","state":["exists","up"]},{"osd":7,"uuid":"9003b8d2-e669-42bd-a24d-a556d4db2cf3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":3612990772},{"type":"v1","addr":"172.21.15.104:6825","nonce":3612990772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":3612990772},{"type":"v1","addr":"172.21.15.104:6827","nonce":3612990772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":3612990772},{"type":"v1","addr":"172.21.15.104:6831","nonce":3612990772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":3612990772},{"type":"v1","addr":"172.21.15.104:6829","nonce":3612990772}]},"public_addr":"172.21.15.104:6825/3612990772","cluster_addr":"172.21.15.104:6827/3612990772","heartbeat_back_addr":"172.21.15.104:6831/3612990772","heartbeat_front_addr":"172.21.15.104:6829/3612990772","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T12:25:33.048636+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T12:25:54.292231+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T12:26:15.498328+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T12:26:37.596296+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T12:26:58.051100+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T12:27:21.613879+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T12:27:44.820201+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T12:28:08.730029+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.79:0/4127214407":"2024-09-22T12:23:45.405786+0000","172.21.15.79:0/3564175523":"2024-09-22T12:24:10.760847+0000","172.21.15.79:0/1473746868":"2024-09-22T12:28:38.322124+0000","172.21.15.79:0/3967018586":"2024-09-22T12:23:45.405786+0000","172.21.15.79:6800/306254740":"2024-09-22T12:28:38.322124+0000","172.21.15.79:0/3136532329":"2024-09-22T12:28:38.322124+0000","172.21.15.79:6800/3138719122":"2024-09-22T12:23:45.405786+0000","172.21.15.79:0/1207296335":"2024-09-22T12:23:45.405786+0000","172.21.15.79:0/1561731591":"2024-09-22T12:24:10.760847+0000","172.21.15.79:0/2089075974":"2024-09-22T12:28:38.322124+0000","172.21.15.79:6801/3138719122":"2024-09-22T12:23:45.405786+0000","172.21.15.79:6800/3479175059":"2024-09-22T12:24:10.760847+0000","172.21.15.79:6801/3479175059":"2024-09-22T12:24:10.760847+0000","172.21.15.79:0/2503007178":"2024-09-22T12:24:10.760847+0000","172.21.15.79:6801/306254740":"2024-09-22T12:28:38.322124+0000","172.21.15.79:0/1823093535":"2024-09-22T12:28:38.322124+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T12:29:04.667 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:29:04 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: [21/Sep/2024:12:29:04] ENGINE Serving on http://:::9283 2024-09-21T12:29:04.668 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:29:04 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: [21/Sep/2024:12:29:04] ENGINE Bus STARTED 2024-09-21T12:29:04.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:29:04 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:04.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:29:04 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:04.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:29:04 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:04.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:29:04 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:04.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:29:04 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:04.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:29:04 smithi079 ceph-mon[23699]: Regenerating cephadm self-signed grafana TLS certificates 2024-09-21T12:29:04.943 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:29:04 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:04.943 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:29:04 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:04.943 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:29:04 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-21T12:29:04.943 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:29:04 smithi079 ceph-mon[23699]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-21T12:29:04.943 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:29:04 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:04.943 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:29:04 smithi079 ceph-mon[23699]: Deploying daemon grafana.a on smithi104 2024-09-21T12:29:04.943 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:29:04 smithi079 ceph-mon[23699]: from='client.? 172.21.15.79:0/3787181816' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-21T12:29:04.943 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:29:04 smithi079 ceph-mon[23699]: pgmap v16: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:29:04.943 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:29:04 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:04.943 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:29:04 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:04.943 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:29:04 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:04.944 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:29:04 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:04.944 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:29:04 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:04.944 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:29:04 smithi079 ceph-mon[30023]: Regenerating cephadm self-signed grafana TLS certificates 2024-09-21T12:29:04.944 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:29:04 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:04.944 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:29:04 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:04.944 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:29:04 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-21T12:29:04.944 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:29:04 smithi079 ceph-mon[30023]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-21T12:29:04.944 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:29:04 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:04.944 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:29:04 smithi079 ceph-mon[30023]: Deploying daemon grafana.a on smithi104 2024-09-21T12:29:04.944 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:29:04 smithi079 ceph-mon[30023]: from='client.? 172.21.15.79:0/3787181816' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-21T12:29:04.944 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:29:04 smithi079 ceph-mon[30023]: pgmap v16: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:29:05.151 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:29:04 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:05.152 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:29:04 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:05.152 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:29:04 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:05.152 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:29:04 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:05.152 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:29:04 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:05.152 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:29:04 smithi104 ceph-mon[25639]: Regenerating cephadm self-signed grafana TLS certificates 2024-09-21T12:29:05.152 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:29:04 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:05.152 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:29:04 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:05.152 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:29:04 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-21T12:29:05.152 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:29:04 smithi104 ceph-mon[25639]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-21T12:29:05.152 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:29:04 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:05.152 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:29:04 smithi104 ceph-mon[25639]: Deploying daemon grafana.a on smithi104 2024-09-21T12:29:05.152 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:29:04 smithi104 ceph-mon[25639]: from='client.? 172.21.15.79:0/3787181816' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-21T12:29:05.152 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:29:04 smithi104 ceph-mon[25639]: pgmap v16: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:29:05.690 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-09-21T12:29:05.690 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 33168022-7814-11ef-baf6-efdc52797490 -- ceph osd dump --format=json 2024-09-21T12:29:06.655 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:29:07.667 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:29:07 smithi104 ceph-mon[25639]: pgmap v17: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:29:07.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:29:07 smithi079 ceph-mon[23699]: pgmap v17: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:29:07.692 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:29:07 smithi079 ceph-mon[30023]: pgmap v17: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:29:08.644 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-21T12:29:08.644 INFO:teuthology.orchestra.run.smithi079.stdout:{"epoch":59,"fsid":"33168022-7814-11ef-baf6-efdc52797490","created":"2024-09-21T12:23:21.896565+0000","modified":"2024-09-21T12:28:38.322154+0000","last_up_change":"2024-09-21T12:28:09.822191+0000","last_in_change":"2024-09-21T12:27:52.435732+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T12:26:19.078594+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"aa2c4408-1b95-4d23-9f60-d73b204290db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6802","nonce":846480130},{"type":"v1","addr":"172.21.15.79:6803","nonce":846480130}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6804","nonce":846480130},{"type":"v1","addr":"172.21.15.79:6805","nonce":846480130}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6808","nonce":846480130},{"type":"v1","addr":"172.21.15.79:6809","nonce":846480130}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6806","nonce":846480130},{"type":"v1","addr":"172.21.15.79:6807","nonce":846480130}]},"public_addr":"172.21.15.79:6803/846480130","cluster_addr":"172.21.15.79:6805/846480130","heartbeat_back_addr":"172.21.15.79:6809/846480130","heartbeat_front_addr":"172.21.15.79:6807/846480130","state":["exists","up"]},{"osd":1,"uuid":"a05b4be6-79b2-4dde-bd51-14cafd833c5a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6810","nonce":1764722078},{"type":"v1","addr":"172.21.15.79:6811","nonce":1764722078}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6812","nonce":1764722078},{"type":"v1","addr":"172.21.15.79:6813","nonce":1764722078}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6816","nonce":1764722078},{"type":"v1","addr":"172.21.15.79:6817","nonce":1764722078}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6814","nonce":1764722078},{"type":"v1","addr":"172.21.15.79:6815","nonce":1764722078}]},"public_addr":"172.21.15.79:6811/1764722078","cluster_addr":"172.21.15.79:6813/1764722078","heartbeat_back_addr":"172.21.15.79:6817/1764722078","heartbeat_front_addr":"172.21.15.79:6815/1764722078","state":["exists","up"]},{"osd":2,"uuid":"8b9ddbf7-8264-4e90-bcf0-e63a7f5e7e9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6818","nonce":2982011382},{"type":"v1","addr":"172.21.15.79:6819","nonce":2982011382}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6820","nonce":2982011382},{"type":"v1","addr":"172.21.15.79:6821","nonce":2982011382}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6824","nonce":2982011382},{"type":"v1","addr":"172.21.15.79:6825","nonce":2982011382}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6822","nonce":2982011382},{"type":"v1","addr":"172.21.15.79:6823","nonce":2982011382}]},"public_addr":"172.21.15.79:6819/2982011382","cluster_addr":"172.21.15.79:6821/2982011382","heartbeat_back_addr":"172.21.15.79:6825/2982011382","heartbeat_front_addr":"172.21.15.79:6823/2982011382","state":["exists","up"]},{"osd":3,"uuid":"65ac97f0-572a-400f-977b-4c20cdc0fc9f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6826","nonce":2894016442},{"type":"v1","addr":"172.21.15.79:6827","nonce":2894016442}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6828","nonce":2894016442},{"type":"v1","addr":"172.21.15.79:6829","nonce":2894016442}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6832","nonce":2894016442},{"type":"v1","addr":"172.21.15.79:6833","nonce":2894016442}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.79:6830","nonce":2894016442},{"type":"v1","addr":"172.21.15.79:6831","nonce":2894016442}]},"public_addr":"172.21.15.79:6827/2894016442","cluster_addr":"172.21.15.79:6829/2894016442","heartbeat_back_addr":"172.21.15.79:6833/2894016442","heartbeat_front_addr":"172.21.15.79:6831/2894016442","state":["exists","up"]},{"osd":4,"uuid":"278156d6-8fd6-4852-9de2-1e11ea7e8917","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6800","nonce":3796154758},{"type":"v1","addr":"172.21.15.104:6801","nonce":3796154758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6802","nonce":3796154758},{"type":"v1","addr":"172.21.15.104:6803","nonce":3796154758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6806","nonce":3796154758},{"type":"v1","addr":"172.21.15.104:6807","nonce":3796154758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6804","nonce":3796154758},{"type":"v1","addr":"172.21.15.104:6805","nonce":3796154758}]},"public_addr":"172.21.15.104:6801/3796154758","cluster_addr":"172.21.15.104:6803/3796154758","heartbeat_back_addr":"172.21.15.104:6807/3796154758","heartbeat_front_addr":"172.21.15.104:6805/3796154758","state":["exists","up"]},{"osd":5,"uuid":"5435eabf-f4d1-4c86-b334-fa3acc9162c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6808","nonce":189336043},{"type":"v1","addr":"172.21.15.104:6809","nonce":189336043}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6810","nonce":189336043},{"type":"v1","addr":"172.21.15.104:6811","nonce":189336043}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6814","nonce":189336043},{"type":"v1","addr":"172.21.15.104:6815","nonce":189336043}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6812","nonce":189336043},{"type":"v1","addr":"172.21.15.104:6813","nonce":189336043}]},"public_addr":"172.21.15.104:6809/189336043","cluster_addr":"172.21.15.104:6811/189336043","heartbeat_back_addr":"172.21.15.104:6815/189336043","heartbeat_front_addr":"172.21.15.104:6813/189336043","state":["exists","up"]},{"osd":6,"uuid":"df36e40d-a08e-4b0f-8774-4862209d0ba0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6816","nonce":2903175260},{"type":"v1","addr":"172.21.15.104:6817","nonce":2903175260}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6818","nonce":2903175260},{"type":"v1","addr":"172.21.15.104:6819","nonce":2903175260}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6822","nonce":2903175260},{"type":"v1","addr":"172.21.15.104:6823","nonce":2903175260}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6820","nonce":2903175260},{"type":"v1","addr":"172.21.15.104:6821","nonce":2903175260}]},"public_addr":"172.21.15.104:6817/2903175260","cluster_addr":"172.21.15.104:6819/2903175260","heartbeat_back_addr":"172.21.15.104:6823/2903175260","heartbeat_front_addr":"172.21.15.104:6821/2903175260","state":["exists","up"]},{"osd":7,"uuid":"9003b8d2-e669-42bd-a24d-a556d4db2cf3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6824","nonce":3612990772},{"type":"v1","addr":"172.21.15.104:6825","nonce":3612990772}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6826","nonce":3612990772},{"type":"v1","addr":"172.21.15.104:6827","nonce":3612990772}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6830","nonce":3612990772},{"type":"v1","addr":"172.21.15.104:6831","nonce":3612990772}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.104:6828","nonce":3612990772},{"type":"v1","addr":"172.21.15.104:6829","nonce":3612990772}]},"public_addr":"172.21.15.104:6825/3612990772","cluster_addr":"172.21.15.104:6827/3612990772","heartbeat_back_addr":"172.21.15.104:6831/3612990772","heartbeat_front_addr":"172.21.15.104:6829/3612990772","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T12:25:33.048636+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T12:25:54.292231+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T12:26:15.498328+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T12:26:37.596296+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T12:26:58.051100+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T12:27:21.613879+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T12:27:44.820201+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T12:28:08.730029+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.79:0/4127214407":"2024-09-22T12:23:45.405786+0000","172.21.15.79:0/3564175523":"2024-09-22T12:24:10.760847+0000","172.21.15.79:0/1473746868":"2024-09-22T12:28:38.322124+0000","172.21.15.79:0/3967018586":"2024-09-22T12:23:45.405786+0000","172.21.15.79:6800/306254740":"2024-09-22T12:28:38.322124+0000","172.21.15.79:0/3136532329":"2024-09-22T12:28:38.322124+0000","172.21.15.79:6800/3138719122":"2024-09-22T12:23:45.405786+0000","172.21.15.79:0/1207296335":"2024-09-22T12:23:45.405786+0000","172.21.15.79:0/1561731591":"2024-09-22T12:24:10.760847+0000","172.21.15.79:0/2089075974":"2024-09-22T12:28:38.322124+0000","172.21.15.79:6801/3138719122":"2024-09-22T12:23:45.405786+0000","172.21.15.79:6800/3479175059":"2024-09-22T12:24:10.760847+0000","172.21.15.79:6801/3479175059":"2024-09-22T12:24:10.760847+0000","172.21.15.79:0/2503007178":"2024-09-22T12:24:10.760847+0000","172.21.15.79:6801/306254740":"2024-09-22T12:28:38.322124+0000","172.21.15.79:0/1823093535":"2024-09-22T12:28:38.322124+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T12:29:08.654 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:29:08 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T12:29:08.654 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:29:08 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T12:29:08.676 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:29:08 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T12:29:09.385 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 33168022-7814-11ef-baf6-efdc52797490 -- ceph tell osd.0 flush_pg_stats 2024-09-21T12:29:09.385 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 33168022-7814-11ef-baf6-efdc52797490 -- ceph tell osd.1 flush_pg_stats 2024-09-21T12:29:09.386 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 33168022-7814-11ef-baf6-efdc52797490 -- ceph tell osd.2 flush_pg_stats 2024-09-21T12:29:09.386 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 33168022-7814-11ef-baf6-efdc52797490 -- ceph tell osd.3 flush_pg_stats 2024-09-21T12:29:09.386 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 33168022-7814-11ef-baf6-efdc52797490 -- ceph tell osd.4 flush_pg_stats 2024-09-21T12:29:09.387 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 33168022-7814-11ef-baf6-efdc52797490 -- ceph tell osd.5 flush_pg_stats 2024-09-21T12:29:09.387 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 33168022-7814-11ef-baf6-efdc52797490 -- ceph tell osd.6 flush_pg_stats 2024-09-21T12:29:09.388 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 33168022-7814-11ef-baf6-efdc52797490 -- ceph tell osd.7 flush_pg_stats 2024-09-21T12:29:09.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:29:09 smithi104 ceph-mon[25639]: pgmap v18: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:29:09.750 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:29:09 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:09.750 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:29:09 smithi104 ceph-mon[25639]: from='client.? 172.21.15.79:0/1063855280' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-21T12:29:09.750 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:29:09 smithi079 ceph-mon[23699]: pgmap v18: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:29:09.750 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:29:09 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:09.750 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:29:09 smithi079 ceph-mon[23699]: from='client.? 172.21.15.79:0/1063855280' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-21T12:29:09.751 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:29:09 smithi079 ceph-mon[30023]: pgmap v18: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:29:09.751 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:29:09 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:09.751 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:29:09 smithi079 ceph-mon[30023]: from='client.? 172.21.15.79:0/1063855280' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-21T12:29:09.783 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:29:09.784 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:29:09.784 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:29:09.788 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:29:09.788 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:29:09.789 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:29:09.792 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:29:09.808 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:29:11.676 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:29:11 smithi104 ceph-mon[25639]: pgmap v19: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:29:11.691 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:29:11 smithi079 ceph-mon[23699]: pgmap v19: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:29:11.692 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:29:11 smithi079 ceph-mon[30023]: pgmap v19: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:29:12.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:29:12 smithi079 ceph-mon[23699]: pgmap v20: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:29:12.942 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:29:12 smithi079 ceph-mon[30023]: pgmap v20: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:29:13.176 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:29:12 smithi104 ceph-mon[25639]: pgmap v20: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:29:13.205 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:29:13 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: ::ffff:172.21.15.104 - - [21/Sep/2024:12:29:13] "GET /metrics HTTP/1.1" 200 26203 "" "Prometheus/2.43.0" 2024-09-21T12:29:15.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:29:15 smithi079 ceph-mon[23699]: pgmap v21: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:29:15.442 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:29:15 smithi079 ceph-mon[30023]: pgmap v21: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:29:15.489 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:29:15 smithi104 ceph-mon[25639]: pgmap v21: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:29:17.426 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:29:17 smithi104 ceph-mon[25639]: pgmap v22: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:29:17.584 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:29:17 smithi079 ceph-mon[23699]: pgmap v22: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:29:17.585 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:29:17 smithi079 ceph-mon[30023]: pgmap v22: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:29:17.839 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:17 smithi104 systemd[1]: Starting Ceph grafana.a for 33168022-7814-11ef-baf6-efdc52797490... 2024-09-21T12:29:17.839 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:17 smithi104 podman[47547]: 2024-09-21 12:29:17.739450219 +0000 UTC m=+0.014855226 image pull 954c08fa618887fe9a6fe572ae541197c77e625b05f4193df7898f5ac649ada9 quay.io/ceph/ceph-grafana:9.4.7 2024-09-21T12:29:18.104 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:17 smithi104 podman[47547]: 2024-09-21 12:29:17.862721569 +0000 UTC m=+0.138126609 container create c85dd02f65193c80f3b7974b4ec16525c6f44314101dce6eee0eb10464c1d55c (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a, vendor=Red Hat, Inc., architecture=x86_64, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=, version=8.5, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.buildah.version=1.32.0, summary=Grafana Container configured for Ceph mgr/dashboard integration, description=Ceph Grafana Container, com.redhat.component=ubi8-container, vcs-type=git, maintainer=Paul Cuzner , io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=Red Hat Universal Base Image 8, name=ubi8, io.openshift.tags=base rhel8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, release=239.1651231664, distribution-scope=public, build-date=2022-04-29T11:48:11.055897) 2024-09-21T12:29:18.104 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 podman[47547]: 2024-09-21 12:29:18.103345166 +0000 UTC m=+0.378750179 container init c85dd02f65193c80f3b7974b4ec16525c6f44314101dce6eee0eb10464c1d55c (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=, vendor=Red Hat, Inc., version=8.5, maintainer=Paul Cuzner , io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., description=Ceph Grafana Container, io.openshift.tags=base rhel8, build-date=2022-04-29T11:48:11.055897, io.buildah.version=1.32.0, release=239.1651231664, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, vcs-type=git, com.redhat.component=ubi8-container, architecture=x86_64, io.k8s.display-name=Red Hat Universal Base Image 8, distribution-scope=public, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, summary=Grafana Container configured for Ceph mgr/dashboard integration, name=ubi8) 2024-09-21T12:29:18.355 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 podman[47547]: 2024-09-21 12:29:18.106277506 +0000 UTC m=+0.381682503 container start c85dd02f65193c80f3b7974b4ec16525c6f44314101dce6eee0eb10464c1d55c (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a, release=239.1651231664, distribution-scope=public, summary=Grafana Container configured for Ceph mgr/dashboard integration, vendor=Red Hat, Inc., vcs-type=git, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, maintainer=Paul Cuzner , build-date=2022-04-29T11:48:11.055897, io.buildah.version=1.32.0, description=Ceph Grafana Container, com.redhat.component=ubi8-container, io.k8s.display-name=Red Hat Universal Base Image 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, name=ubi8, architecture=x86_64, io.openshift.expose-services=, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., version=8.5, io.openshift.tags=base rhel8) 2024-09-21T12:29:18.355 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 bash[47547]: c85dd02f65193c80f3b7974b4ec16525c6f44314101dce6eee0eb10464c1d55c 2024-09-21T12:29:18.355 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 systemd[1]: Started Ceph grafana.a for 33168022-7814-11ef-baf6-efdc52797490. 2024-09-21T12:29:18.355 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=settings t=2024-09-21T12:29:18.259794551Z level=info msg="Starting Grafana" version=9.4.7 commit=4add91f03d branch=HEAD compiled=2023-03-16T23:56:52Z 2024-09-21T12:29:18.355 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=settings t=2024-09-21T12:29:18.25998519Z level=warn msg="\"sentry\" frontend logging provider is deprecated and will be removed in the next major version. Use \"grafana\" provider instead." 2024-09-21T12:29:18.355 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=settings t=2024-09-21T12:29:18.26000454Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini 2024-09-21T12:29:18.355 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=settings t=2024-09-21T12:29:18.260011427Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini 2024-09-21T12:29:18.355 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=settings t=2024-09-21T12:29:18.260017364Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana" 2024-09-21T12:29:18.356 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=settings t=2024-09-21T12:29:18.26002342Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana" 2024-09-21T12:29:18.356 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=settings t=2024-09-21T12:29:18.260029045Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-09-21T12:29:18.356 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=settings t=2024-09-21T12:29:18.260034691Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-09-21T12:29:18.356 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=settings t=2024-09-21T12:29:18.260040465Z level=info msg="Path Home" path=/usr/share/grafana 2024-09-21T12:29:18.356 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=settings t=2024-09-21T12:29:18.260051027Z level=info msg="Path Data" path=/var/lib/grafana 2024-09-21T12:29:18.356 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=settings t=2024-09-21T12:29:18.260056506Z level=info msg="Path Logs" path=/var/log/grafana 2024-09-21T12:29:18.357 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=settings t=2024-09-21T12:29:18.260062235Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins 2024-09-21T12:29:18.357 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=settings t=2024-09-21T12:29:18.26006818Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning 2024-09-21T12:29:18.357 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=settings t=2024-09-21T12:29:18.26007488Z level=info msg="App mode production" 2024-09-21T12:29:18.357 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=sqlstore t=2024-09-21T12:29:18.260123663Z level=info msg="Connecting to DB" dbtype=sqlite3 2024-09-21T12:29:18.357 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=sqlstore t=2024-09-21T12:29:18.260138925Z level=warn msg="SQLite database file has broader permissions than it should" path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-09-21T12:29:18.357 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.260980081Z level=info msg="Starting DB migrations" 2024-09-21T12:29:18.357 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.261058154Z level=info msg="Executing migration" id="create migration_log table" 2024-09-21T12:29:18.357 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.261572131Z level=info msg="Executing migration" id="create user table" 2024-09-21T12:29:18.357 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.262023353Z level=info msg="Executing migration" id="add unique index user.login" 2024-09-21T12:29:18.357 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.262509124Z level=info msg="Executing migration" id="add unique index user.email" 2024-09-21T12:29:18.357 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.26541187Z level=info msg="Executing migration" id="drop index UQE_user_login - v1" 2024-09-21T12:29:18.357 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.26820324Z level=info msg="Executing migration" id="drop index UQE_user_email - v1" 2024-09-21T12:29:18.357 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.27108451Z level=info msg="Executing migration" id="Rename table user to user_v1 - v1" 2024-09-21T12:29:18.357 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.27388672Z level=info msg="Executing migration" id="create user table v2" 2024-09-21T12:29:18.357 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.276637077Z level=info msg="Executing migration" id="create index UQE_user_login - v2" 2024-09-21T12:29:18.358 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.279591865Z level=info msg="Executing migration" id="create index UQE_user_email - v2" 2024-09-21T12:29:18.358 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.282613863Z level=info msg="Executing migration" id="copy data_source v1 to v2" 2024-09-21T12:29:18.358 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.285497597Z level=info msg="Executing migration" id="Drop old table user_v1" 2024-09-21T12:29:18.358 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.288238366Z level=info msg="Executing migration" id="Add column help_flags1 to user table" 2024-09-21T12:29:18.358 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.288709242Z level=info msg="Executing migration" id="Update user table charset" 2024-09-21T12:29:18.358 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.291558569Z level=info msg="Executing migration" id="Add last_seen_at column to user" 2024-09-21T12:29:18.358 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.294431073Z level=info msg="Executing migration" id="Add missing user data" 2024-09-21T12:29:18.358 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.2972882Z level=info msg="Executing migration" id="Add is_disabled column to user" 2024-09-21T12:29:18.358 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.297847365Z level=info msg="Executing migration" id="Add index user.login/user.email" 2024-09-21T12:29:18.358 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.300729295Z level=info msg="Executing migration" id="Add is_service_account column to user" 2024-09-21T12:29:18.358 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.303383434Z level=info msg="Executing migration" id="Update is_service_account column to nullable" 2024-09-21T12:29:18.358 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.306196923Z level=info msg="Executing migration" id="create temp user table v1-7" 2024-09-21T12:29:18.358 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.307027295Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v1-7" 2024-09-21T12:29:18.358 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.309715299Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v1-7" 2024-09-21T12:29:18.358 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.312453845Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v1-7" 2024-09-21T12:29:18.359 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.315195876Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v1-7" 2024-09-21T12:29:18.359 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.317857839Z level=info msg="Executing migration" id="Update temp_user table charset" 2024-09-21T12:29:18.359 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.320524525Z level=info msg="Executing migration" id="drop index IDX_temp_user_email - v1" 2024-09-21T12:29:18.359 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.323279186Z level=info msg="Executing migration" id="drop index IDX_temp_user_org_id - v1" 2024-09-21T12:29:18.359 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.325882381Z level=info msg="Executing migration" id="drop index IDX_temp_user_code - v1" 2024-09-21T12:29:18.359 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.328636371Z level=info msg="Executing migration" id="drop index IDX_temp_user_status - v1" 2024-09-21T12:29:18.359 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.331357403Z level=info msg="Executing migration" id="Rename table temp_user to temp_user_tmp_qwerty - v1" 2024-09-21T12:29:18.359 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.334137916Z level=info msg="Executing migration" id="create temp_user v2" 2024-09-21T12:29:18.359 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.336975223Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v2" 2024-09-21T12:29:18.359 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.339556107Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v2" 2024-09-21T12:29:18.359 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.340531542Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v2" 2024-09-21T12:29:18.359 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.341300589Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v2" 2024-09-21T12:29:18.359 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.342247921Z level=info msg="Executing migration" id="copy temp_user v1 to v2" 2024-09-21T12:29:18.359 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.342957387Z level=info msg="Executing migration" id="drop temp_user_tmp_qwerty" 2024-09-21T12:29:18.360 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.343311592Z level=info msg="Executing migration" id="Set created for temp users that will otherwise prematurely expire" 2024-09-21T12:29:18.360 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.344219821Z level=info msg="Executing migration" id="create star table" 2024-09-21T12:29:18.360 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.344746904Z level=info msg="Executing migration" id="add unique index star.user_id_dashboard_id" 2024-09-21T12:29:18.360 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.345174987Z level=info msg="Executing migration" id="create org table v1" 2024-09-21T12:29:18.360 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.34603127Z level=info msg="Executing migration" id="create index UQE_org_name - v1" 2024-09-21T12:29:18.360 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.346882973Z level=info msg="Executing migration" id="create org_user table v1" 2024-09-21T12:29:18.360 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.347722844Z level=info msg="Executing migration" id="create index IDX_org_user_org_id - v1" 2024-09-21T12:29:18.360 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.348708564Z level=info msg="Executing migration" id="create index UQE_org_user_org_id_user_id - v1" 2024-09-21T12:29:18.360 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.349214768Z level=info msg="Executing migration" id="create index IDX_org_user_user_id - v1" 2024-09-21T12:29:18.360 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.350165142Z level=info msg="Executing migration" id="Update org table charset" 2024-09-21T12:29:18.360 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.350899658Z level=info msg="Executing migration" id="Update org_user table charset" 2024-09-21T12:29:18.360 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.351835495Z level=info msg="Executing migration" id="Migrate all Read Only Viewers to Viewers" 2024-09-21T12:29:18.360 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.352304289Z level=info msg="Executing migration" id="create dashboard table" 2024-09-21T12:29:18.360 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.353136281Z level=info msg="Executing migration" id="add index dashboard.account_id" 2024-09-21T12:29:18.360 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.354023068Z level=info msg="Executing migration" id="add unique index dashboard_account_id_slug" 2024-09-21T12:29:18.361 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.354848647Z level=info msg="Executing migration" id="create dashboard_tag table" 2024-09-21T12:29:18.361 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.355253127Z level=info msg="Executing migration" id="add unique index dashboard_tag.dasboard_id_term" 2024-09-21T12:29:18.607 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.356087224Z level=info msg="Executing migration" id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2024-09-21T12:29:18.607 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.357053524Z level=info msg="Executing migration" id="Rename table dashboard to dashboard_v1 - v1" 2024-09-21T12:29:18.607 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.357943074Z level=info msg="Executing migration" id="create dashboard v2" 2024-09-21T12:29:18.607 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.358341226Z level=info msg="Executing migration" id="create index IDX_dashboard_org_id - v2" 2024-09-21T12:29:18.607 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.359187321Z level=info msg="Executing migration" id="create index UQE_dashboard_org_id_slug - v2" 2024-09-21T12:29:18.607 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.360037599Z level=info msg="Executing migration" id="copy dashboard v1 to v2" 2024-09-21T12:29:18.607 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.360883198Z level=info msg="Executing migration" id="drop table dashboard_v1" 2024-09-21T12:29:18.607 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.361301412Z level=info msg="Executing migration" id="alter dashboard.data to mediumtext v1" 2024-09-21T12:29:18.607 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.362209837Z level=info msg="Executing migration" id="Add column updated_by in dashboard - v2" 2024-09-21T12:29:18.608 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.362878968Z level=info msg="Executing migration" id="Add column created_by in dashboard - v2" 2024-09-21T12:29:18.608 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.363378339Z level=info msg="Executing migration" id="Add column gnetId in dashboard" 2024-09-21T12:29:18.608 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.364289045Z level=info msg="Executing migration" id="Add index for gnetId in dashboard" 2024-09-21T12:29:18.608 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.364902527Z level=info msg="Executing migration" id="Add column plugin_id in dashboard" 2024-09-21T12:29:18.608 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.365394131Z level=info msg="Executing migration" id="Add index for plugin_id in dashboard" 2024-09-21T12:29:18.608 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.368180018Z level=info msg="Executing migration" id="Add index for dashboard_id in dashboard_tag" 2024-09-21T12:29:18.608 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.368725379Z level=info msg="Executing migration" id="Update dashboard table charset" 2024-09-21T12:29:18.608 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.371090222Z level=info msg="Executing migration" id="Update dashboard_tag table charset" 2024-09-21T12:29:18.608 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.373808184Z level=info msg="Executing migration" id="Add column folder_id in dashboard" 2024-09-21T12:29:18.608 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.374410884Z level=info msg="Executing migration" id="Add column isFolder in dashboard" 2024-09-21T12:29:18.608 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.377376432Z level=info msg="Executing migration" id="Add column has_acl in dashboard" 2024-09-21T12:29:18.608 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.378069406Z level=info msg="Executing migration" id="Add column uid in dashboard" 2024-09-21T12:29:18.608 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.380814233Z level=info msg="Executing migration" id="Update uid column values in dashboard" 2024-09-21T12:29:18.609 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.382989176Z level=info msg="Executing migration" id="Add unique index dashboard_org_id_uid" 2024-09-21T12:29:18.609 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.38342164Z level=info msg="Executing migration" id="Remove unique index org_id_slug" 2024-09-21T12:29:18.609 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.386246852Z level=info msg="Executing migration" id="Update dashboard title length" 2024-09-21T12:29:18.609 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.386553449Z level=info msg="Executing migration" id="Add unique index for dashboard_org_id_title_folder_id" 2024-09-21T12:29:18.609 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.389376601Z level=info msg="Executing migration" id="create dashboard_provisioning" 2024-09-21T12:29:18.609 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.389836897Z level=info msg="Executing migration" id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2024-09-21T12:29:18.609 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.392027931Z level=info msg="Executing migration" id="create dashboard_provisioning v2" 2024-09-21T12:29:18.609 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.392453166Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2024-09-21T12:29:18.609 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.395425294Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2024-09-21T12:29:18.609 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.398092009Z level=info msg="Executing migration" id="copy dashboard_provisioning v1 to v2" 2024-09-21T12:29:18.609 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.400700218Z level=info msg="Executing migration" id="drop dashboard_provisioning_tmp_qwerty" 2024-09-21T12:29:18.609 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.401105305Z level=info msg="Executing migration" id="Add check_sum column" 2024-09-21T12:29:18.609 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.40378239Z level=info msg="Executing migration" id="Add index for dashboard_title" 2024-09-21T12:29:18.610 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.406542005Z level=info msg="Executing migration" id="delete tags for deleted dashboards" 2024-09-21T12:29:18.610 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.409185564Z level=info msg="Executing migration" id="delete stars for deleted dashboards" 2024-09-21T12:29:18.610 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.409535512Z level=info msg="Executing migration" id="Add index for dashboard_is_folder" 2024-09-21T12:29:18.611 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.412149446Z level=info msg="Executing migration" id="Add isPublic for dashboard" 2024-09-21T12:29:18.611 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.412902833Z level=info msg="Executing migration" id="create data_source table" 2024-09-21T12:29:18.611 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.413765643Z level=info msg="Executing migration" id="add index data_source.account_id" 2024-09-21T12:29:18.611 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.414636534Z level=info msg="Executing migration" id="add unique index data_source.account_id_name" 2024-09-21T12:29:18.611 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.415207379Z level=info msg="Executing migration" id="drop index IDX_data_source_account_id - v1" 2024-09-21T12:29:18.612 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.416035079Z level=info msg="Executing migration" id="drop index UQE_data_source_account_id_name - v1" 2024-09-21T12:29:18.612 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.416883713Z level=info msg="Executing migration" id="Rename table data_source to data_source_v1 - v1" 2024-09-21T12:29:18.612 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.417948645Z level=info msg="Executing migration" id="create data_source table v2" 2024-09-21T12:29:18.612 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.418814993Z level=info msg="Executing migration" id="create index IDX_data_source_org_id - v2" 2024-09-21T12:29:18.612 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.419803884Z level=info msg="Executing migration" id="create index UQE_data_source_org_id_name - v2" 2024-09-21T12:29:18.612 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.420531407Z level=info msg="Executing migration" id="Drop old table data_source_v1 #2" 2024-09-21T12:29:18.612 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.42128891Z level=info msg="Executing migration" id="Add column with_credentials" 2024-09-21T12:29:18.612 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.421877583Z level=info msg="Executing migration" id="Add secure json data column" 2024-09-21T12:29:18.612 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.422446248Z level=info msg="Executing migration" id="Update data_source table charset" 2024-09-21T12:29:18.612 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.423237701Z level=info msg="Executing migration" id="Update initial version to 1" 2024-09-21T12:29:18.612 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.423578883Z level=info msg="Executing migration" id="Add read_only data column" 2024-09-21T12:29:18.612 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.42413409Z level=info msg="Executing migration" id="Migrate logging ds to loki ds" 2024-09-21T12:29:18.612 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.425095615Z level=info msg="Executing migration" id="Update json_data with nulls" 2024-09-21T12:29:18.612 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.425871759Z level=info msg="Executing migration" id="Add uid column" 2024-09-21T12:29:18.613 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.426828924Z level=info msg="Executing migration" id="Update uid value" 2024-09-21T12:29:18.613 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.427574384Z level=info msg="Executing migration" id="Add unique index datasource_org_id_uid" 2024-09-21T12:29:18.613 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.428544583Z level=info msg="Executing migration" id="add unique index datasource_org_id_is_default" 2024-09-21T12:29:18.613 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.429333207Z level=info msg="Executing migration" id="create api_key table" 2024-09-21T12:29:18.613 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.430153185Z level=info msg="Executing migration" id="add index api_key.account_id" 2024-09-21T12:29:18.613 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.431042195Z level=info msg="Executing migration" id="add index api_key.key" 2024-09-21T12:29:18.613 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.431871316Z level=info msg="Executing migration" id="add index api_key.account_id_name" 2024-09-21T12:29:18.613 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.432875624Z level=info msg="Executing migration" id="drop index IDX_api_key_account_id - v1" 2024-09-21T12:29:18.613 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.433575283Z level=info msg="Executing migration" id="drop index UQE_api_key_key - v1" 2024-09-21T12:29:18.613 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.434376824Z level=info msg="Executing migration" id="drop index UQE_api_key_account_id_name - v1" 2024-09-21T12:29:18.613 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.434810006Z level=info msg="Executing migration" id="Rename table api_key to api_key_v1 - v1" 2024-09-21T12:29:18.613 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.435857427Z level=info msg="Executing migration" id="create api_key table v2" 2024-09-21T12:29:18.613 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.43675086Z level=info msg="Executing migration" id="create index IDX_api_key_org_id - v2" 2024-09-21T12:29:18.613 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.437173139Z level=info msg="Executing migration" id="create index UQE_api_key_key - v2" 2024-09-21T12:29:18.613 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.438148163Z level=info msg="Executing migration" id="create index UQE_api_key_org_id_name - v2" 2024-09-21T12:29:18.614 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.439017313Z level=info msg="Executing migration" id="copy api_key v1 to v2" 2024-09-21T12:29:18.614 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.439349686Z level=info msg="Executing migration" id="Drop old table api_key_v1" 2024-09-21T12:29:18.614 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.439864384Z level=info msg="Executing migration" id="Update api_key table charset" 2024-09-21T12:29:18.614 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.440174748Z level=info msg="Executing migration" id="Add expires to api_key table" 2024-09-21T12:29:18.614 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.440709547Z level=info msg="Executing migration" id="Add service account foreign key" 2024-09-21T12:29:18.614 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.441271407Z level=info msg="Executing migration" id="set service account foreign key to nil if 0" 2024-09-21T12:29:18.614 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.443686089Z level=info msg="Executing migration" id="Add last_used_at to api_key table" 2024-09-21T12:29:18.614 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.446379715Z level=info msg="Executing migration" id="Add is_revoked column to api_key table" 2024-09-21T12:29:18.614 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.447127847Z level=info msg="Executing migration" id="create dashboard_snapshot table v4" 2024-09-21T12:29:18.614 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.449978349Z level=info msg="Executing migration" id="drop table dashboard_snapshot_v4 #1" 2024-09-21T12:29:18.614 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.452979148Z level=info msg="Executing migration" id="create dashboard_snapshot table v5 #2" 2024-09-21T12:29:18.614 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.455683732Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_key - v5" 2024-09-21T12:29:18.614 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.456127845Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_delete_key - v5" 2024-09-21T12:29:18.614 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.459073288Z level=info msg="Executing migration" id="create index IDX_dashboard_snapshot_user_id - v5" 2024-09-21T12:29:18.614 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.461750983Z level=info msg="Executing migration" id="alter dashboard_snapshot to mediumtext v2" 2024-09-21T12:29:18.614 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.464684099Z level=info msg="Executing migration" id="Update dashboard_snapshot table charset" 2024-09-21T12:29:18.615 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.464992823Z level=info msg="Executing migration" id="Add column external_delete_url to dashboard_snapshots table" 2024-09-21T12:29:18.615 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.468179713Z level=info msg="Executing migration" id="Add encrypted dashboard json column" 2024-09-21T12:29:18.615 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.471091752Z level=info msg="Executing migration" id="Change dashboard_encrypted column to MEDIUMBLOB" 2024-09-21T12:29:18.615 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.473778871Z level=info msg="Executing migration" id="create quota table v1" 2024-09-21T12:29:18.615 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.474267203Z level=info msg="Executing migration" id="create index UQE_quota_org_id_user_id_target - v1" 2024-09-21T12:29:18.615 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.477044175Z level=info msg="Executing migration" id="Update quota table charset" 2024-09-21T12:29:18.615 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.479700283Z level=info msg="Executing migration" id="create plugin_setting table" 2024-09-21T12:29:18.615 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.480132304Z level=info msg="Executing migration" id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2024-09-21T12:29:18.615 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.480556222Z level=info msg="Executing migration" id="Add column plugin_version to plugin_settings" 2024-09-21T12:29:18.615 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.483129815Z level=info msg="Executing migration" id="Update plugin_setting table charset" 2024-09-21T12:29:18.615 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.485655865Z level=info msg="Executing migration" id="create session table" 2024-09-21T12:29:18.615 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.486210385Z level=info msg="Executing migration" id="Drop old table playlist table" 2024-09-21T12:29:18.615 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.486543628Z level=info msg="Executing migration" id="Drop old table playlist_item table" 2024-09-21T12:29:18.615 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.486824386Z level=info msg="Executing migration" id="create playlist table v2" 2024-09-21T12:29:18.615 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.48725036Z level=info msg="Executing migration" id="create playlist item table v2" 2024-09-21T12:29:18.616 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.48999727Z level=info msg="Executing migration" id="Update playlist table charset" 2024-09-21T12:29:18.616 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.490405075Z level=info msg="Executing migration" id="Update playlist_item table charset" 2024-09-21T12:29:18.616 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.490772254Z level=info msg="Executing migration" id="drop preferences table v2" 2024-09-21T12:29:18.616 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.491873954Z level=info msg="Executing migration" id="drop preferences table v3" 2024-09-21T12:29:18.616 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.492202698Z level=info msg="Executing migration" id="create preferences table v3" 2024-09-21T12:29:18.616 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.492606403Z level=info msg="Executing migration" id="Update preferences table charset" 2024-09-21T12:29:18.616 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.492877385Z level=info msg="Executing migration" id="Add column team_id in preferences" 2024-09-21T12:29:18.616 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.495381818Z level=info msg="Executing migration" id="Update team_id column values in preferences" 2024-09-21T12:29:18.616 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.495708946Z level=info msg="Executing migration" id="Add column week_start in preferences" 2024-09-21T12:29:18.616 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.49630898Z level=info msg="Executing migration" id="Add column preferences.json_data" 2024-09-21T12:29:18.616 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.49891021Z level=info msg="Executing migration" id="alter preferences.json_data to mediumtext v1" 2024-09-21T12:29:18.616 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.499226749Z level=info msg="Executing migration" id="Add preferences index org_id" 2024-09-21T12:29:18.616 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.501700141Z level=info msg="Executing migration" id="Add preferences index user_id" 2024-09-21T12:29:18.616 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.504268275Z level=info msg="Executing migration" id="create alert table v1" 2024-09-21T12:29:18.616 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.507053244Z level=info msg="Executing migration" id="add index alert org_id & id " 2024-09-21T12:29:18.616 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.509742887Z level=info msg="Executing migration" id="add index alert state" 2024-09-21T12:29:18.617 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.510293953Z level=info msg="Executing migration" id="add index alert dashboard_id" 2024-09-21T12:29:18.617 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.510720956Z level=info msg="Executing migration" id="Create alert_rule_tag table v1" 2024-09-21T12:29:18.617 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.511111252Z level=info msg="Executing migration" id="Add unique index alert_rule_tag.alert_id_tag_id" 2024-09-21T12:29:18.617 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.511564338Z level=info msg="Executing migration" id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" 2024-09-21T12:29:18.617 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.511947423Z level=info msg="Executing migration" id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" 2024-09-21T12:29:18.617 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.513395935Z level=info msg="Executing migration" id="Create alert_rule_tag table v2" 2024-09-21T12:29:18.617 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.513801673Z level=info msg="Executing migration" id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" 2024-09-21T12:29:18.617 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.514269321Z level=info msg="Executing migration" id="copy alert_rule_tag v1 to v2" 2024-09-21T12:29:18.617 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.514597933Z level=info msg="Executing migration" id="drop table alert_rule_tag_v1" 2024-09-21T12:29:18.617 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.514937642Z level=info msg="Executing migration" id="create alert_notification table v1" 2024-09-21T12:29:18.617 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.515380777Z level=info msg="Executing migration" id="Add column is_default" 2024-09-21T12:29:18.617 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.51600444Z level=info msg="Executing migration" id="Add column frequency" 2024-09-21T12:29:18.617 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.516621232Z level=info msg="Executing migration" id="Add column send_reminder" 2024-09-21T12:29:18.617 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.517367267Z level=info msg="Executing migration" id="Add column disable_resolve_message" 2024-09-21T12:29:18.617 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.51805283Z level=info msg="Executing migration" id="add index alert_notification org_id & name" 2024-09-21T12:29:18.618 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.518483509Z level=info msg="Executing migration" id="Update alert table charset" 2024-09-21T12:29:18.618 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.518743015Z level=info msg="Executing migration" id="Update alert_notification table charset" 2024-09-21T12:29:18.618 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.519038009Z level=info msg="Executing migration" id="create notification_journal table v1" 2024-09-21T12:29:18.618 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.519423544Z level=info msg="Executing migration" id="add index notification_journal org_id & alert_id & notifier_id" 2024-09-21T12:29:18.618 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.519839489Z level=info msg="Executing migration" id="drop alert_notification_journal" 2024-09-21T12:29:18.618 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.520226517Z level=info msg="Executing migration" id="create alert_notification_state table v1" 2024-09-21T12:29:18.618 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.520583235Z level=info msg="Executing migration" id="add index alert_notification_state org_id & alert_id & notifier_id" 2024-09-21T12:29:18.618 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.520968539Z level=info msg="Executing migration" id="Add for to alert table" 2024-09-21T12:29:18.618 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.521583252Z level=info msg="Executing migration" id="Add column uid in alert_notification" 2024-09-21T12:29:18.618 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.522212106Z level=info msg="Executing migration" id="Update uid column values in alert_notification" 2024-09-21T12:29:18.618 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.522523406Z level=info msg="Executing migration" id="Add unique index alert_notification_org_id_uid" 2024-09-21T12:29:18.618 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.522939898Z level=info msg="Executing migration" id="Remove unique index org_id_name" 2024-09-21T12:29:18.618 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.523331952Z level=info msg="Executing migration" id="Add column secure_settings in alert_notification" 2024-09-21T12:29:18.618 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.523916771Z level=info msg="Executing migration" id="alter alert.settings to mediumtext" 2024-09-21T12:29:18.618 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.524215075Z level=info msg="Executing migration" id="Add non-unique index alert_notification_state_alert_id" 2024-09-21T12:29:18.618 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.52458839Z level=info msg="Executing migration" id="Add non-unique index alert_rule_tag_alert_id" 2024-09-21T12:29:18.619 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.52498153Z level=info msg="Executing migration" id="Drop old annotation table v4" 2024-09-21T12:29:18.619 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.525298405Z level=info msg="Executing migration" id="create annotation table v5" 2024-09-21T12:29:18.619 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.525716298Z level=info msg="Executing migration" id="add index annotation 0 v3" 2024-09-21T12:29:18.619 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.526150726Z level=info msg="Executing migration" id="add index annotation 1 v3" 2024-09-21T12:29:18.619 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.526548689Z level=info msg="Executing migration" id="add index annotation 2 v3" 2024-09-21T12:29:18.619 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.526959825Z level=info msg="Executing migration" id="add index annotation 3 v3" 2024-09-21T12:29:18.619 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.52860861Z level=info msg="Executing migration" id="add index annotation 4 v3" 2024-09-21T12:29:18.619 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.530975925Z level=info msg="Executing migration" id="Update annotation table charset" 2024-09-21T12:29:18.619 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.531330373Z level=info msg="Executing migration" id="Add column region_id to annotation table" 2024-09-21T12:29:18.619 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.531946894Z level=info msg="Executing migration" id="Drop category_id index" 2024-09-21T12:29:18.619 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.5323602Z level=info msg="Executing migration" id="Add column tags to annotation table" 2024-09-21T12:29:18.619 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.533006888Z level=info msg="Executing migration" id="Create annotation_tag table v2" 2024-09-21T12:29:18.620 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.533863797Z level=info msg="Executing migration" id="Add unique index annotation_tag.annotation_id_tag_id" 2024-09-21T12:29:18.620 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.5343035Z level=info msg="Executing migration" id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" 2024-09-21T12:29:18.620 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.535120711Z level=info msg="Executing migration" id="Rename table annotation_tag to annotation_tag_v2 - v2" 2024-09-21T12:29:18.620 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.536679809Z level=info msg="Executing migration" id="Create annotation_tag table v3" 2024-09-21T12:29:18.620 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.537082188Z level=info msg="Executing migration" id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" 2024-09-21T12:29:18.620 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.537950583Z level=info msg="Executing migration" id="copy annotation_tag v2 to v3" 2024-09-21T12:29:18.620 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.538278148Z level=info msg="Executing migration" id="drop table annotation_tag_v2" 2024-09-21T12:29:18.620 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.539091587Z level=info msg="Executing migration" id="Update alert annotations and set TEXT to empty" 2024-09-21T12:29:18.620 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.539443334Z level=info msg="Executing migration" id="Add created time to annotation table" 2024-09-21T12:29:18.620 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.540107229Z level=info msg="Executing migration" id="Add updated time to annotation table" 2024-09-21T12:29:18.620 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.54095306Z level=info msg="Executing migration" id="Add index for created in annotation table" 2024-09-21T12:29:18.620 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.541374495Z level=info msg="Executing migration" id="Add index for updated in annotation table" 2024-09-21T12:29:18.620 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.542231166Z level=info msg="Executing migration" id="Convert existing annotations from seconds to milliseconds" 2024-09-21T12:29:18.620 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.543071846Z level=info msg="Executing migration" id="Add epoch_end column" 2024-09-21T12:29:18.621 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.543943068Z level=info msg="Executing migration" id="Add index for epoch_end" 2024-09-21T12:29:18.621 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.544423475Z level=info msg="Executing migration" id="Make epoch_end the same as epoch" 2024-09-21T12:29:18.621 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.547146377Z level=info msg="Executing migration" id="Move region to single row" 2024-09-21T12:29:18.621 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.547535801Z level=info msg="Executing migration" id="Remove index org_id_epoch from annotation table" 2024-09-21T12:29:18.621 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.549873049Z level=info msg="Executing migration" id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2024-09-21T12:29:18.621 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.550339341Z level=info msg="Executing migration" id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2024-09-21T12:29:18.621 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.552584739Z level=info msg="Executing migration" id="Add index for org_id_epoch_end_epoch on annotation table" 2024-09-21T12:29:18.621 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.553074832Z level=info msg="Executing migration" id="Remove index org_id_epoch_epoch_end from annotation table" 2024-09-21T12:29:18.621 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.553499977Z level=info msg="Executing migration" id="Add index for alert_id on annotation table" 2024-09-21T12:29:18.621 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.556297282Z level=info msg="Executing migration" id="Increase tags column to length 4096" 2024-09-21T12:29:18.621 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.556837157Z level=info msg="Executing migration" id="create test_data table" 2024-09-21T12:29:18.621 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.559075054Z level=info msg="Executing migration" id="create dashboard_version table v1" 2024-09-21T12:29:18.621 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.561711673Z level=info msg="Executing migration" id="add index dashboard_version.dashboard_id" 2024-09-21T12:29:18.622 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.562196594Z level=info msg="Executing migration" id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2024-09-21T12:29:18.622 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.563088457Z level=info msg="Executing migration" id="Set dashboard version to 1 where 0" 2024-09-21T12:29:18.622 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.564053704Z level=info msg="Executing migration" id="save existing dashboard data in dashboard_version table v1" 2024-09-21T12:29:18.622 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.564930605Z level=info msg="Executing migration" id="alter dashboard_version.data to mediumtext v1" 2024-09-21T12:29:18.622 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.565385864Z level=info msg="Executing migration" id="create team table" 2024-09-21T12:29:18.622 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.56622203Z level=info msg="Executing migration" id="add index team.org_id" 2024-09-21T12:29:18.622 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.56707121Z level=info msg="Executing migration" id="add unique index team_org_id_name" 2024-09-21T12:29:18.622 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.568073719Z level=info msg="Executing migration" id="create team member table" 2024-09-21T12:29:18.622 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.568661755Z level=info msg="Executing migration" id="add index team_member.org_id" 2024-09-21T12:29:18.622 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.569599497Z level=info msg="Executing migration" id="add unique index team_member_org_id_team_id_user_id" 2024-09-21T12:29:18.622 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.570363685Z level=info msg="Executing migration" id="add index team_member.team_id" 2024-09-21T12:29:18.622 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.571363125Z level=info msg="Executing migration" id="Add column email to team table" 2024-09-21T12:29:18.622 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.572263679Z level=info msg="Executing migration" id="Add column external to team_member table" 2024-09-21T12:29:18.622 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.580392811Z level=info msg="Executing migration" id="Add column permission to team_member table" 2024-09-21T12:29:18.623 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.581390073Z level=info msg="Executing migration" id="create dashboard acl table" 2024-09-21T12:29:18.623 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.582222168Z level=info msg="Executing migration" id="add index dashboard_acl_dashboard_id" 2024-09-21T12:29:18.623 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.582688877Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_user_id" 2024-09-21T12:29:18.623 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.583562097Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_team_id" 2024-09-21T12:29:18.623 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.584522161Z level=info msg="Executing migration" id="add index dashboard_acl_user_id" 2024-09-21T12:29:18.623 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.585406997Z level=info msg="Executing migration" id="add index dashboard_acl_team_id" 2024-09-21T12:29:18.623 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.585822904Z level=info msg="Executing migration" id="add index dashboard_acl_org_id_role" 2024-09-21T12:29:18.623 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.586827747Z level=info msg="Executing migration" id="add index dashboard_permission" 2024-09-21T12:29:18.623 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.587603328Z level=info msg="Executing migration" id="save default acl rules in dashboard_acl table" 2024-09-21T12:29:18.623 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.588498613Z level=info msg="Executing migration" id="delete acl rules for deleted dashboards and folders" 2024-09-21T12:29:18.623 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.589487543Z level=info msg="Executing migration" id="create tag table" 2024-09-21T12:29:18.623 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.5904014Z level=info msg="Executing migration" id="add index tag.key_value" 2024-09-21T12:29:18.623 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.591260491Z level=info msg="Executing migration" id="create login attempt table" 2024-09-21T12:29:18.623 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.592078852Z level=info msg="Executing migration" id="add index login_attempt.username" 2024-09-21T12:29:18.624 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.593148658Z level=info msg="Executing migration" id="drop index IDX_login_attempt_username - v1" 2024-09-21T12:29:18.624 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.594079755Z level=info msg="Executing migration" id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2024-09-21T12:29:18.624 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.596251176Z level=info msg="Executing migration" id="create login_attempt v2" 2024-09-21T12:29:18.624 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.597183543Z level=info msg="Executing migration" id="create index IDX_login_attempt_username - v2" 2024-09-21T12:29:18.624 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.597976234Z level=info msg="Executing migration" id="copy login_attempt v1 to v2" 2024-09-21T12:29:18.624 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.59899054Z level=info msg="Executing migration" id="drop login_attempt_tmp_qwerty" 2024-09-21T12:29:18.624 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.599772016Z level=info msg="Executing migration" id="create user auth table" 2024-09-21T12:29:18.624 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.602704013Z level=info msg="Executing migration" id="create index IDX_user_auth_auth_module_auth_id - v1" 2024-09-21T12:29:18.624 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.605616148Z level=info msg="Executing migration" id="alter user_auth.auth_id to length 190" 2024-09-21T12:29:18.624 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.608395177Z level=info msg="Executing migration" id="Add OAuth access token to user_auth" 2024-09-21T12:29:18.859 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.611461894Z level=info msg="Executing migration" id="Add OAuth refresh token to user_auth" 2024-09-21T12:29:18.859 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.614370664Z level=info msg="Executing migration" id="Add OAuth token type to user_auth" 2024-09-21T12:29:18.859 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.61717505Z level=info msg="Executing migration" id="Add OAuth expiry to user_auth" 2024-09-21T12:29:18.859 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.62016429Z level=info msg="Executing migration" id="Add index to user_id column in user_auth" 2024-09-21T12:29:18.859 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.622988806Z level=info msg="Executing migration" id="Add OAuth ID token to user_auth" 2024-09-21T12:29:18.859 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.625800552Z level=info msg="Executing migration" id="create server_lock table" 2024-09-21T12:29:18.859 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.628685002Z level=info msg="Executing migration" id="add index server_lock.operation_uid" 2024-09-21T12:29:18.859 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.631656249Z level=info msg="Executing migration" id="create user auth token table" 2024-09-21T12:29:18.859 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.634755264Z level=info msg="Executing migration" id="add unique index user_auth_token.auth_token" 2024-09-21T12:29:18.859 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.637750654Z level=info msg="Executing migration" id="add unique index user_auth_token.prev_auth_token" 2024-09-21T12:29:18.859 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.640707113Z level=info msg="Executing migration" id="add index user_auth_token.user_id" 2024-09-21T12:29:18.860 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.643617454Z level=info msg="Executing migration" id="Add revoked_at to the user auth token" 2024-09-21T12:29:18.860 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.646618674Z level=info msg="Executing migration" id="create cache_data table" 2024-09-21T12:29:18.860 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.649352272Z level=info msg="Executing migration" id="add unique index cache_data.cache_key" 2024-09-21T12:29:18.860 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.652068837Z level=info msg="Executing migration" id="create short_url table v1" 2024-09-21T12:29:18.860 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.654894678Z level=info msg="Executing migration" id="add index short_url.org_id-uid" 2024-09-21T12:29:18.860 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.65758912Z level=info msg="Executing migration" id="alter table short_url alter column created_by type to bigint" 2024-09-21T12:29:18.860 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.66027671Z level=info msg="Executing migration" id="delete alert_definition table" 2024-09-21T12:29:18.860 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.66299205Z level=info msg="Executing migration" id="recreate alert_definition table" 2024-09-21T12:29:18.861 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.665764389Z level=info msg="Executing migration" id="add index in alert_definition on org_id and title columns" 2024-09-21T12:29:18.861 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.668579596Z level=info msg="Executing migration" id="add index in alert_definition on org_id and uid columns" 2024-09-21T12:29:18.861 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.671178305Z level=info msg="Executing migration" id="alter alert_definition table data column to mediumtext in mysql" 2024-09-21T12:29:18.861 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.673975859Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and title columns" 2024-09-21T12:29:18.861 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.676725684Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and uid columns" 2024-09-21T12:29:18.861 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.679530948Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and title columns" 2024-09-21T12:29:18.861 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.682213325Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and uid columns" 2024-09-21T12:29:18.861 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.684937214Z level=info msg="Executing migration" id="Add column paused in alert_definition" 2024-09-21T12:29:18.861 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.687645881Z level=info msg="Executing migration" id="drop alert_definition table" 2024-09-21T12:29:18.862 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.690229864Z level=info msg="Executing migration" id="delete alert_definition_version table" 2024-09-21T12:29:18.862 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.692857797Z level=info msg="Executing migration" id="recreate alert_definition_version table" 2024-09-21T12:29:18.862 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.695641071Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_id and version columns" 2024-09-21T12:29:18.862 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.698368464Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_uid and version columns" 2024-09-21T12:29:18.862 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.699132891Z level=info msg="Executing migration" id="alter alert_definition_version table data column to mediumtext in mysql" 2024-09-21T12:29:18.862 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.700101824Z level=info msg="Executing migration" id="drop alert_definition_version table" 2024-09-21T12:29:18.862 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.700822868Z level=info msg="Executing migration" id="create alert_instance table" 2024-09-21T12:29:18.862 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.701812229Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, def_uid and current_state columns" 2024-09-21T12:29:18.862 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.702554947Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, current_state columns" 2024-09-21T12:29:18.862 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.703513647Z level=info msg="Executing migration" id="add column current_state_end to alert_instance" 2024-09-21T12:29:18.862 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.704328137Z level=info msg="Executing migration" id="remove index def_org_id, def_uid, current_state on alert_instance" 2024-09-21T12:29:18.862 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.705183021Z level=info msg="Executing migration" id="remove index def_org_id, current_state on alert_instance" 2024-09-21T12:29:18.862 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.706040613Z level=info msg="Executing migration" id="rename def_org_id to rule_org_id in alert_instance" 2024-09-21T12:29:18.862 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.709480849Z level=info msg="Executing migration" id="rename def_uid to rule_uid in alert_instance" 2024-09-21T12:29:18.863 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.71282762Z level=info msg="Executing migration" id="add index rule_org_id, rule_uid, current_state on alert_instance" 2024-09-21T12:29:18.863 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.713703706Z level=info msg="Executing migration" id="add index rule_org_id, current_state on alert_instance" 2024-09-21T12:29:18.863 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.714680052Z level=info msg="Executing migration" id="add current_reason column related to current_state" 2024-09-21T12:29:18.863 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.715681678Z level=info msg="Executing migration" id="create alert_rule table" 2024-09-21T12:29:18.863 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.716685651Z level=info msg="Executing migration" id="add index in alert_rule on org_id and title columns" 2024-09-21T12:29:18.863 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.717373506Z level=info msg="Executing migration" id="add index in alert_rule on org_id and uid columns" 2024-09-21T12:29:18.863 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.718403363Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespace_uid, group_uid columns" 2024-09-21T12:29:18.863 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.71908193Z level=info msg="Executing migration" id="alter alert_rule table data column to mediumtext in mysql" 2024-09-21T12:29:18.863 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.720075733Z level=info msg="Executing migration" id="add column for to alert_rule" 2024-09-21T12:29:18.863 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.721002554Z level=info msg="Executing migration" id="add column annotations to alert_rule" 2024-09-21T12:29:18.863 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.721848344Z level=info msg="Executing migration" id="add column labels to alert_rule" 2024-09-21T12:29:18.863 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.722696761Z level=info msg="Executing migration" id="remove unique index from alert_rule on org_id, title columns" 2024-09-21T12:29:18.863 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.723700865Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespase_uid and title columns" 2024-09-21T12:29:18.863 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.72443525Z level=info msg="Executing migration" id="add dashboard_uid column to alert_rule" 2024-09-21T12:29:18.863 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.725288939Z level=info msg="Executing migration" id="add panel_id column to alert_rule" 2024-09-21T12:29:18.864 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.726142214Z level=info msg="Executing migration" id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" 2024-09-21T12:29:18.864 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.727025694Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule" 2024-09-21T12:29:18.864 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.727886657Z level=info msg="Executing migration" id="add is_paused column to alert_rule table" 2024-09-21T12:29:18.864 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.728755627Z level=info msg="Executing migration" id="fix is_paused column for alert_rule table" 2024-09-21T12:29:18.864 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.729614392Z level=info msg="Executing migration" id="create alert_rule_version table" 2024-09-21T12:29:18.864 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.730597427Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" 2024-09-21T12:29:18.864 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.731307202Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" 2024-09-21T12:29:18.864 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.732355398Z level=info msg="Executing migration" id="alter alert_rule_version table data column to mediumtext in mysql" 2024-09-21T12:29:18.864 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.733085521Z level=info msg="Executing migration" id="add column for to alert_rule_version" 2024-09-21T12:29:18.864 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.733993983Z level=info msg="Executing migration" id="add column annotations to alert_rule_version" 2024-09-21T12:29:18.864 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.736864568Z level=info msg="Executing migration" id="add column labels to alert_rule_version" 2024-09-21T12:29:18.864 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.739959288Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule_version" 2024-09-21T12:29:18.864 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.742970405Z level=info msg="Executing migration" id="add is_paused column to alert_rule_versions table" 2024-09-21T12:29:18.864 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.745840337Z level=info msg="Executing migration" id="fix is_paused column for alert_rule_version table" 2024-09-21T12:29:18.865 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.749046801Z level=info msg="Executing migration" id=create_alert_configuration_table 2024-09-21T12:29:18.865 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.751966571Z level=info msg="Executing migration" id="Add column default in alert_configuration" 2024-09-21T12:29:18.865 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.754845084Z level=info msg="Executing migration" id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" 2024-09-21T12:29:18.865 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.757919958Z level=info msg="Executing migration" id="add column org_id in alert_configuration" 2024-09-21T12:29:18.865 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.762834312Z level=info msg="Executing migration" id="add index in alert_configuration table on org_id column" 2024-09-21T12:29:18.865 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.763775962Z level=info msg="Executing migration" id="add configuration_hash column to alert_configuration" 2024-09-21T12:29:18.865 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.766835156Z level=info msg="Executing migration" id=create_ngalert_configuration_table 2024-09-21T12:29:18.865 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.770003067Z level=info msg="Executing migration" id="add index in ngalert_configuration on org_id column" 2024-09-21T12:29:18.865 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.772901776Z level=info msg="Executing migration" id="add column send_alerts_to in ngalert_configuration" 2024-09-21T12:29:18.865 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.775853495Z level=info msg="Executing migration" id="create provenance_type table" 2024-09-21T12:29:18.865 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.778782512Z level=info msg="Executing migration" id="add index to uniquify (record_key, record_type, org_id) columns" 2024-09-21T12:29:18.865 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.77928459Z level=info msg="Executing migration" id="create alert_image table" 2024-09-21T12:29:18.865 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.780283852Z level=info msg="Executing migration" id="add unique index on token to alert_image table" 2024-09-21T12:29:18.865 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.781011004Z level=info msg="Executing migration" id="support longer URLs in alert_image table" 2024-09-21T12:29:18.866 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.783810532Z level=info msg="Executing migration" id=create_alert_configuration_history_table 2024-09-21T12:29:18.866 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.786605925Z level=info msg="Executing migration" id="drop non-unique orgID index on alert_configuration" 2024-09-21T12:29:18.866 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.787079108Z level=info msg="Executing migration" id="drop unique orgID index on alert_configuration if exists" 2024-09-21T12:29:18.866 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.787158601Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop unique orgID index on alert_configuration if exists" 2024-09-21T12:29:18.866 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.790029601Z level=info msg="Executing migration" id="extract alertmanager configuration history to separate table" 2024-09-21T12:29:18.866 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.79300193Z level=info msg="Executing migration" id="add unique index on orgID to alert_configuration" 2024-09-21T12:29:18.866 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.795722541Z level=info msg="Executing migration" id="clear migration entry \"remove unified alerting data\"" 2024-09-21T12:29:18.866 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.795857248Z level=info msg="Executing migration" id="move dashboard alerts to unified alerting" 2024-09-21T12:29:18.866 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.795971794Z level=info msg="alerts found to migrate" alerts=0 2024-09-21T12:29:18.866 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.798599778Z level=info msg="Executing migration" id="create library_element table v1" 2024-09-21T12:29:18.866 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.801423422Z level=info msg="Executing migration" id="add index library_element org_id-folder_id-name-kind" 2024-09-21T12:29:18.866 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.804155544Z level=info msg="Executing migration" id="create library_element_connection table v1" 2024-09-21T12:29:18.866 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.806958191Z level=info msg="Executing migration" id="add index library_element_connection element_id-kind-connection_id" 2024-09-21T12:29:18.866 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.809766534Z level=info msg="Executing migration" id="add unique index library_element org_id_uid" 2024-09-21T12:29:18.867 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.813169664Z level=info msg="Executing migration" id="increase max description length to 2048" 2024-09-21T12:29:18.867 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.816134888Z level=info msg="Executing migration" id="clone move dashboard alerts to unified alerting" 2024-09-21T12:29:18.867 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.819049463Z level=info msg="Executing migration" id="create data_keys table" 2024-09-21T12:29:18.867 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.822053367Z level=info msg="Executing migration" id="create secrets table" 2024-09-21T12:29:18.867 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.824923637Z level=info msg="Executing migration" id="rename data_keys name column to id" 2024-09-21T12:29:18.867 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.830141539Z level=info msg="Executing migration" id="add name column into data_keys" 2024-09-21T12:29:18.867 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.833012351Z level=info msg="Executing migration" id="copy data_keys id column values into name" 2024-09-21T12:29:18.867 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.835723938Z level=info msg="Executing migration" id="rename data_keys name column to label" 2024-09-21T12:29:18.867 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.840161413Z level=info msg="Executing migration" id="rename data_keys id column back to name" 2024-09-21T12:29:18.867 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.84434968Z level=info msg="Executing migration" id="create kv_store table v1" 2024-09-21T12:29:18.867 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.847042466Z level=info msg="Executing migration" id="add index kv_store.org_id-namespace-key" 2024-09-21T12:29:18.867 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.849622015Z level=info msg="Executing migration" id="update dashboard_uid and panel_id from existing annotations" 2024-09-21T12:29:18.867 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.852350322Z level=info msg="Executing migration" id="create permission table" 2024-09-21T12:29:18.867 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.85515069Z level=info msg="Executing migration" id="add unique index permission.role_id" 2024-09-21T12:29:18.867 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.857899321Z level=info msg="Executing migration" id="add unique index role_id_action_scope" 2024-09-21T12:29:19.144 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.860483351Z level=info msg="Executing migration" id="create role table" 2024-09-21T12:29:19.144 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.863026735Z level=info msg="Executing migration" id="add column display_name" 2024-09-21T12:29:19.144 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.865663027Z level=info msg="Executing migration" id="add column group_name" 2024-09-21T12:29:19.144 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.868284889Z level=info msg="Executing migration" id="add index role.org_id" 2024-09-21T12:29:19.144 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.871108213Z level=info msg="Executing migration" id="add unique index role_org_id_name" 2024-09-21T12:29:19.144 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.873727569Z level=info msg="Executing migration" id="add index role_org_id_uid" 2024-09-21T12:29:19.144 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.876347156Z level=info msg="Executing migration" id="create team role table" 2024-09-21T12:29:19.145 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.879058788Z level=info msg="Executing migration" id="add index team_role.org_id" 2024-09-21T12:29:19.145 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.881762739Z level=info msg="Executing migration" id="add unique index team_role_org_id_team_id_role_id" 2024-09-21T12:29:19.145 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.88418238Z level=info msg="Executing migration" id="add index team_role.team_id" 2024-09-21T12:29:19.145 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.885075444Z level=info msg="Executing migration" id="create user role table" 2024-09-21T12:29:19.145 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.88581244Z level=info msg="Executing migration" id="add index user_role.org_id" 2024-09-21T12:29:19.145 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.886772903Z level=info msg="Executing migration" id="add unique index user_role_org_id_user_id_role_id" 2024-09-21T12:29:19.145 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.887517304Z level=info msg="Executing migration" id="add index user_role.user_id" 2024-09-21T12:29:19.145 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.888372672Z level=info msg="Executing migration" id="create builtin role table" 2024-09-21T12:29:19.145 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.889224465Z level=info msg="Executing migration" id="add index builtin_role.role_id" 2024-09-21T12:29:19.145 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.890201381Z level=info msg="Executing migration" id="add index builtin_role.name" 2024-09-21T12:29:19.145 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.890934124Z level=info msg="Executing migration" id="Add column org_id to builtin_role table" 2024-09-21T12:29:19.145 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.891896068Z level=info msg="Executing migration" id="add index builtin_role.org_id" 2024-09-21T12:29:19.146 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.892427283Z level=info msg="Executing migration" id="add unique index builtin_role_org_id_role_id_role" 2024-09-21T12:29:19.146 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.893220682Z level=info msg="Executing migration" id="Remove unique index role_org_id_uid" 2024-09-21T12:29:19.146 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.893833577Z level=info msg="Executing migration" id="add unique index role.uid" 2024-09-21T12:29:19.146 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.894738578Z level=info msg="Executing migration" id="create seed assignment table" 2024-09-21T12:29:19.146 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.895234223Z level=info msg="Executing migration" id="add unique index builtin_role_role_name" 2024-09-21T12:29:19.146 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.896028102Z level=info msg="Executing migration" id="add column hidden to role table" 2024-09-21T12:29:19.146 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.897185904Z level=info msg="Executing migration" id="create query_history table v1" 2024-09-21T12:29:19.146 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.89762079Z level=info msg="Executing migration" id="add index query_history.org_id-created_by-datasource_uid" 2024-09-21T12:29:19.147 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.898090549Z level=info msg="Executing migration" id="alter table query_history alter column created_by type to bigint" 2024-09-21T12:29:19.147 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.898403775Z level=info msg="Executing migration" id="teams permissions migration" 2024-09-21T12:29:19.147 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.898807894Z level=info msg="Executing migration" id="dashboard permissions" 2024-09-21T12:29:19.147 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.899324092Z level=info msg="Executing migration" id="dashboard permissions uid scopes" 2024-09-21T12:29:19.147 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.899820081Z level=info msg="Executing migration" id="drop managed folder create actions" 2024-09-21T12:29:19.147 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.900171977Z level=info msg="Executing migration" id="alerting notification permissions" 2024-09-21T12:29:19.147 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.900528147Z level=info msg="Executing migration" id="create query_history_star table v1" 2024-09-21T12:29:19.147 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.901008975Z level=info msg="Executing migration" id="add index query_history.user_id-query_uid" 2024-09-21T12:29:19.147 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.901626285Z level=info msg="Executing migration" id="add column org_id in query_history_star" 2024-09-21T12:29:19.147 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.902781996Z level=info msg="Executing migration" id="alter table query_history_star_mig column user_id type to bigint" 2024-09-21T12:29:19.147 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.903125573Z level=info msg="Executing migration" id="create correlation table v1" 2024-09-21T12:29:19.147 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.903566111Z level=info msg="Executing migration" id="add index correlations.uid" 2024-09-21T12:29:19.147 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.904031666Z level=info msg="Executing migration" id="add index correlations.source_uid" 2024-09-21T12:29:19.147 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.904521117Z level=info msg="Executing migration" id="add correlation config column" 2024-09-21T12:29:19.148 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.905547334Z level=info msg="Executing migration" id="create entity_events table" 2024-09-21T12:29:19.148 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.905980939Z level=info msg="Executing migration" id="create dashboard public config v1" 2024-09-21T12:29:19.148 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.906777005Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v1" 2024-09-21T12:29:19.148 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.906873402Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index UQE_dashboard_public_config_uid - v1" 2024-09-21T12:29:19.148 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.907189767Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-09-21T12:29:19.148 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.907297308Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-09-21T12:29:19.148 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.907596334Z level=info msg="Executing migration" id="Drop old dashboard public config table" 2024-09-21T12:29:19.148 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.908015288Z level=info msg="Executing migration" id="recreate dashboard public config v1" 2024-09-21T12:29:19.148 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.908446517Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v1" 2024-09-21T12:29:19.148 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.908904078Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-09-21T12:29:19.148 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.909367195Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v2" 2024-09-21T12:29:19.148 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.909790942Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-09-21T12:29:19.148 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.910238486Z level=info msg="Executing migration" id="Drop public config table" 2024-09-21T12:29:19.148 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.910723511Z level=info msg="Executing migration" id="Recreate dashboard public config v2" 2024-09-21T12:29:19.148 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.911215052Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v2" 2024-09-21T12:29:19.149 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.911666633Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-09-21T12:29:19.149 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.912157356Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_access_token - v2" 2024-09-21T12:29:19.149 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.91297102Z level=info msg="Executing migration" id="Rename table dashboard_public_config to dashboard_public - v2" 2024-09-21T12:29:19.149 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.916130202Z level=info msg="Executing migration" id="add annotations_enabled column" 2024-09-21T12:29:19.149 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.917212483Z level=info msg="Executing migration" id="add time_selection_enabled column" 2024-09-21T12:29:19.149 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.918268478Z level=info msg="Executing migration" id="delete orphaned public dashboards" 2024-09-21T12:29:19.149 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.918609656Z level=info msg="Executing migration" id="add share column" 2024-09-21T12:29:19.149 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.919807296Z level=info msg="Executing migration" id="create default alerting folders" 2024-09-21T12:29:19.149 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.920288544Z level=info msg="Executing migration" id="create file table" 2024-09-21T12:29:19.149 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.920737004Z level=info msg="Executing migration" id="file table idx: path natural pk" 2024-09-21T12:29:19.149 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.921539687Z level=info msg="Executing migration" id="file table idx: parent_folder_path_hash fast folder retrieval" 2024-09-21T12:29:19.149 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.922048625Z level=info msg="Executing migration" id="create file_meta table" 2024-09-21T12:29:19.149 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.922463188Z level=info msg="Executing migration" id="file table idx: path key" 2024-09-21T12:29:19.149 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.923309513Z level=info msg="Executing migration" id="set path collation in file table" 2024-09-21T12:29:19.149 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.92414405Z level=info msg="Executing migration" id="managed permissions migration" 2024-09-21T12:29:19.150 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.924985372Z level=info msg="Executing migration" id="managed folder permissions alert actions migration" 2024-09-21T12:29:19.150 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.925332515Z level=info msg="Executing migration" id="RBAC action name migrator" 2024-09-21T12:29:19.150 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.926161948Z level=info msg="Executing migration" id="Add UID column to playlist" 2024-09-21T12:29:19.150 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.927202876Z level=info msg="Executing migration" id="Update uid column values in playlist" 2024-09-21T12:29:19.150 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.927540927Z level=info msg="Executing migration" id="Add index for uid in playlist" 2024-09-21T12:29:19.150 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.928485586Z level=info msg="Executing migration" id="update group index for alert rules" 2024-09-21T12:29:19.150 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.928919501Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated migration" 2024-09-21T12:29:19.150 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.929751954Z level=info msg="Executing migration" id="admin only folder/dashboard permission" 2024-09-21T12:29:19.151 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.930602937Z level=info msg="Executing migration" id="add action column to seed_assignment" 2024-09-21T12:29:19.151 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.931637053Z level=info msg="Executing migration" id="add scope column to seed_assignment" 2024-09-21T12:29:19.151 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.932666759Z level=info msg="Executing migration" id="remove unique index builtin_role_role_name before nullable update" 2024-09-21T12:29:19.151 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.933539189Z level=info msg="Executing migration" id="update seed_assignment role_name column to nullable" 2024-09-21T12:29:19.151 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.943493085Z level=info msg="Executing migration" id="add unique index builtin_role_name back" 2024-09-21T12:29:19.151 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.944361682Z level=info msg="Executing migration" id="add unique index builtin_role_action_scope" 2024-09-21T12:29:19.151 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.944833342Z level=info msg="Executing migration" id="add primary key to seed_assigment" 2024-09-21T12:29:19.151 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.948250177Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated fixed migration" 2024-09-21T12:29:19.151 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.949141502Z level=info msg="Executing migration" id="migrate external alertmanagers to datsourcse" 2024-09-21T12:29:19.151 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.94997538Z level=info msg="Executing migration" id="create folder table" 2024-09-21T12:29:19.151 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.950496693Z level=info msg="Executing migration" id="Add index for parent_uid" 2024-09-21T12:29:19.151 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.953541474Z level=info msg="Executing migration" id="Add unique index for folder.uid and folder.org_id" 2024-09-21T12:29:19.151 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.956330313Z level=info msg="Executing migration" id="Update folder title length" 2024-09-21T12:29:19.151 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.959069933Z level=info msg="Executing migration" id="Add unique index for folder.title and folder.parent_uid" 2024-09-21T12:29:19.151 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=migrator t=2024-09-21T12:29:18.962092895Z level=info msg="migrations completed" performed=485 skipped=0 duration=701.05209ms 2024-09-21T12:29:19.152 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:18 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=sqlstore t=2024-09-21T12:29:18.962663696Z level=info msg="Created default organization" 2024-09-21T12:29:19.152 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:19 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=plugin.loader t=2024-09-21T12:29:19.020112212Z level=info msg="Plugin registered" pluginID=input 2024-09-21T12:29:19.152 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:19 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=plugin.loader t=2024-09-21T12:29:19.049103531Z level=info msg="Plugin registered" pluginID=vonage-status-panel 2024-09-21T12:29:19.152 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:19 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=plugin.loader t=2024-09-21T12:29:19.049140055Z level=info msg="Plugin registered" pluginID=grafana-piechart-panel 2024-09-21T12:29:19.152 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:19 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=secrets t=2024-09-21T12:29:19.049178621Z level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1 2024-09-21T12:29:19.152 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:19 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=query_data t=2024-09-21T12:29:19.050847474Z level=info msg="Query Service initialization" 2024-09-21T12:29:19.152 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:19 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=live.push_http t=2024-09-21T12:29:19.054014944Z level=info msg="Live Push Gateway initialization" 2024-09-21T12:29:19.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:29:19 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:19.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:29:19 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:19.428 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:29:19 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:19.428 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:29:19 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:19.428 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:29:19 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:29:19.428 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:29:19 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi079", "caps": []}]: dispatch 2024-09-21T12:29:19.428 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:29:19 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi104", "caps": []}]: dispatch 2024-09-21T12:29:19.428 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:29:19 smithi104 ceph-mon[25639]: pgmap v23: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:29:19.428 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:29:19 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:19.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:29:19 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:19.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:29:19 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:19.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:29:19 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:19.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:29:19 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:19.443 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:29:19 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:29:19.443 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:29:19 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi079", "caps": []}]: dispatch 2024-09-21T12:29:19.443 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:29:19 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi104", "caps": []}]: dispatch 2024-09-21T12:29:19.443 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:29:19 smithi079 ceph-mon[23699]: pgmap v23: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:29:19.443 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:29:19 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:19.443 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:29:19 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:19.443 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:29:19 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:19.443 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:29:19 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:19.443 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:29:19 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:19.443 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:29:19 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:29:19.443 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:29:19 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi079", "caps": []}]: dispatch 2024-09-21T12:29:19.443 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:29:19 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi104", "caps": []}]: dispatch 2024-09-21T12:29:19.444 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:29:19 smithi079 ceph-mon[30023]: pgmap v23: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:29:19.444 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:29:19 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:19.974 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:19 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=infra.usagestats.collector t=2024-09-21T12:29:19.919266606Z level=info msg="registering usage stat providers" usageStatsProvidersLen=2 2024-09-21T12:29:19.974 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:19 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=provisioning.datasources t=2024-09-21T12:29:19.973284085Z level=info msg="inserting datasource from configuration " name=Dashboard1 uid=P43CA22E17D0F9596 2024-09-21T12:29:20.246 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:20 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=provisioning.datasources t=2024-09-21T12:29:20.005298319Z level=info msg="inserting datasource from configuration " name=Loki uid=P8E80F9AEF21F6940 2024-09-21T12:29:20.247 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:20 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=provisioning.alerting t=2024-09-21T12:29:20.027071449Z level=info msg="starting to provision alerting" 2024-09-21T12:29:20.247 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:20 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=provisioning.alerting t=2024-09-21T12:29:20.027091022Z level=info msg="finished to provision alerting" 2024-09-21T12:29:20.247 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:20 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=grafanaStorageLogger t=2024-09-21T12:29:20.027248168Z level=info msg="storage starting" 2024-09-21T12:29:20.247 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:20 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=ngalert.state.manager t=2024-09-21T12:29:20.027339961Z level=info msg="Warming state cache for startup" 2024-09-21T12:29:20.247 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:20 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=http.server t=2024-09-21T12:29:20.028980565Z level=info msg="HTTP Server Listen" address=[::]:3000 protocol=https subUrl= socket= 2024-09-21T12:29:20.247 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:20 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=ngalert.state.manager t=2024-09-21T12:29:20.037433386Z level=info msg="State cache has been initialized" states=0 duration=10.092883ms 2024-09-21T12:29:20.247 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:20 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=ticker t=2024-09-21T12:29:20.037484594Z level=info msg=starting first_tick=2024-09-21T12:29:30Z 2024-09-21T12:29:20.247 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:20 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=ngalert.multiorg.alertmanager t=2024-09-21T12:29:20.037499127Z level=info msg="starting MultiOrg Alertmanager" 2024-09-21T12:29:20.247 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:20 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=sqlstore.transactions t=2024-09-21T12:29:20.084929116Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-09-21T12:29:20.247 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:20 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=sqlstore.transactions t=2024-09-21T12:29:20.095399719Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=1 code="database is locked" 2024-09-21T12:29:20.247 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:29:20 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a[47562]: logger=sqlstore.transactions t=2024-09-21T12:29:20.13289798Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-09-21T12:29:20.346 INFO:teuthology.orchestra.run.smithi079.stdout:231928234000 2024-09-21T12:29:20.346 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 33168022-7814-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.7 2024-09-21T12:29:20.497 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:29:20 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:20.498 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:29:20 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:20.498 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:29:20 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:29:20.498 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:29:20 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:29:20.498 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:29:20 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:20.498 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:29:20 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi079", "caps": []}]: dispatch 2024-09-21T12:29:20.498 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:29:20 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:20.532 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:29:20 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:20.532 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:29:20 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:20.532 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:29:20 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:29:20.532 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:29:20 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:29:20.532 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:29:20 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:20.532 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:29:20 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi079", "caps": []}]: dispatch 2024-09-21T12:29:20.532 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:29:20 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:20.533 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:29:20 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:20.533 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:29:20 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:20.533 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:29:20 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:29:20.533 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:29:20 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:29:20.533 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:29:20 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:20.533 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:29:20 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi079", "caps": []}]: dispatch 2024-09-21T12:29:20.533 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:29:20 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:21.177 INFO:teuthology.orchestra.run.smithi079.stdout:107374182435 2024-09-21T12:29:21.178 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 33168022-7814-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.3 2024-09-21T12:29:21.400 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:29:21 smithi079 ceph-mon[23699]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-21T12:29:21.400 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:29:21 smithi079 ceph-mon[23699]: Reconfiguring alertmanager.a (dependencies changed)... 2024-09-21T12:29:21.400 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:29:21 smithi079 ceph-mon[23699]: Reconfiguring daemon alertmanager.a on smithi079 2024-09-21T12:29:21.400 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:29:21 smithi079 ceph-mon[23699]: pgmap v24: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:29:21.400 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:29:21 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:21.400 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:29:21 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:21.400 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:29:21 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:21.400 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:29:21 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:21.400 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:29:21 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:21.401 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:29:21 smithi079 ceph-mon[30023]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-21T12:29:21.401 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:29:21 smithi079 ceph-mon[30023]: Reconfiguring alertmanager.a (dependencies changed)... 2024-09-21T12:29:21.401 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:29:21 smithi079 ceph-mon[30023]: Reconfiguring daemon alertmanager.a on smithi079 2024-09-21T12:29:21.401 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:29:21 smithi079 ceph-mon[30023]: pgmap v24: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:29:21.401 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:29:21 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:21.401 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:29:21 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:21.401 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:29:21 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:21.401 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:29:21 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:21.401 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:29:21 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:21.634 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:29:21 smithi104 ceph-mon[25639]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-21T12:29:21.634 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:29:21 smithi104 ceph-mon[25639]: Reconfiguring alertmanager.a (dependencies changed)... 2024-09-21T12:29:21.634 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:29:21 smithi104 ceph-mon[25639]: Reconfiguring daemon alertmanager.a on smithi079 2024-09-21T12:29:21.634 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:29:21 smithi104 ceph-mon[25639]: pgmap v24: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:29:21.634 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:29:21 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:21.634 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:29:21 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:21.634 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:29:21 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:21.635 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:29:21 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:21.635 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:29:21 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:23.372 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:29:23 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: ::ffff:172.21.15.104 - - [21/Sep/2024:12:29:23] "GET /metrics HTTP/1.1" 200 26203 "" "Prometheus/2.43.0" 2024-09-21T12:29:23.519 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:29:23 smithi104 ceph-mon[25639]: pgmap v25: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:29:23.519 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:29:23 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:23.519 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:29:23 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T12:29:23.664 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:29:23 smithi079 ceph-mon[23699]: pgmap v25: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:29:23.664 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:29:23 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:23.664 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:29:23 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T12:29:23.664 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:29:23 smithi079 ceph-mon[30023]: pgmap v25: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:29:23.664 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:29:23 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:23.664 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:29:23 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T12:29:24.884 INFO:teuthology.orchestra.run.smithi079.stdout:34359738415 2024-09-21T12:29:24.885 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 33168022-7814-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.0 2024-09-21T12:29:25.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:29:25 smithi104 ceph-mon[25639]: pgmap v26: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:29:25.691 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:29:25 smithi079 ceph-mon[23699]: pgmap v26: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:29:25.692 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:29:25 smithi079 ceph-mon[30023]: pgmap v26: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:29:26.939 INFO:teuthology.orchestra.run.smithi079.stdout:128849018910 2024-09-21T12:29:26.940 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 33168022-7814-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.4 2024-09-21T12:29:27.353 INFO:teuthology.orchestra.run.smithi079.stdout:197568495638 2024-09-21T12:29:27.354 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 33168022-7814-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.6 2024-09-21T12:29:27.614 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:29:27 smithi079 ceph-mon[23699]: pgmap v27: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:29:27.614 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:29:27 smithi079 ceph-mon[30023]: pgmap v27: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:29:27.676 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:29:27 smithi104 ceph-mon[25639]: pgmap v27: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:29:28.290 INFO:teuthology.orchestra.run.smithi079.stdout:77309411369 2024-09-21T12:29:28.291 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 33168022-7814-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.2 2024-09-21T12:29:29.255 INFO:teuthology.orchestra.run.smithi079.stdout:163208757275 2024-09-21T12:29:29.255 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 33168022-7814-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.5 2024-09-21T12:29:29.629 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:29:29 smithi079 ceph-mon[23699]: pgmap v28: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:29:29.629 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:29:29 smithi079 ceph-mon[30023]: pgmap v28: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:29:29.648 INFO:teuthology.orchestra.run.smithi079.stdout:55834574893 2024-09-21T12:29:29.648 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 33168022-7814-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.1 2024-09-21T12:29:29.676 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:29:29 smithi104 ceph-mon[25639]: pgmap v28: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:29:29.779 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:29:31.185 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Sep 21 12:29:31 smithi079 systemd[1]: Stopping Ceph alertmanager.a for 33168022-7814-11ef-baf6-efdc52797490... 2024-09-21T12:29:31.442 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:29:31 smithi079 ceph-mon[30023]: pgmap v29: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:29:31.442 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:29:31 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:31.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:29:31 smithi079 ceph-mon[23699]: pgmap v29: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:29:31.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:29:31 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:31.618 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:29:31.621 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:29:31.622 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:29:31.622 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:29:31.626 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:29:31.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:29:31 smithi104 ceph-mon[25639]: pgmap v29: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:29:31.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:29:31 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:32.831 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Sep 21 12:29:32 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-alertmanager-a[49549]: ts=2024-09-21T12:29:32.739Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-09-21T12:29:33.091 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Sep 21 12:29:32 smithi079 podman[53715]: 2024-09-21 12:29:32.834176093 +0000 UTC m=+0.384370061 container died 88e00126f964a731b22d0d46d762d8510fddd307155e6af704a19763801b254f (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-33168022-7814-11ef-baf6-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-21T12:29:33.091 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:29:32 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:33.092 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:29:32 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:33.092 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:29:32 smithi079 ceph-mon[23699]: pgmap v30: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:29:33.092 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:29:32 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:33.092 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:29:32 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:33.092 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:29:32 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:33.092 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:29:32 smithi079 ceph-mon[30023]: pgmap v30: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:29:33.092 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:29:32 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:33.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:29:32 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:33.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:29:32 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:33.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:29:32 smithi104 ceph-mon[25639]: pgmap v30: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:29:33.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:29:32 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:33.343 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Sep 21 12:29:33 smithi079 podman[53715]: 2024-09-21 12:29:33.324374767 +0000 UTC m=+0.874568720 container cleanup 88e00126f964a731b22d0d46d762d8510fddd307155e6af704a19763801b254f (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-33168022-7814-11ef-baf6-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-21T12:29:33.343 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Sep 21 12:29:33 smithi079 bash[53715]: ceph-33168022-7814-11ef-baf6-efdc52797490-alertmanager-a 2024-09-21T12:29:33.343 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:29:33 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: ::ffff:172.21.15.104 - - [21/Sep/2024:12:29:33] "GET /metrics HTTP/1.1" 200 26200 "" "Prometheus/2.43.0" 2024-09-21T12:29:33.861 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Sep 21 12:29:33 smithi079 podman[53731]: 2024-09-21 12:29:33.534661379 +0000 UTC m=+0.748610227 container remove 88e00126f964a731b22d0d46d762d8510fddd307155e6af704a19763801b254f (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-33168022-7814-11ef-baf6-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-21T12:29:33.861 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Sep 21 12:29:33 smithi079 podman[53731]: 2024-09-21 12:29:33.626952717 +0000 UTC m=+0.840901568 volume remove 774c12552acd6d77ce3e8704b30a94cf7924221113135ed620099ade892f43a2 2024-09-21T12:29:33.862 INFO:teuthology.orchestra.run.smithi079.stdout:107374182437 2024-09-21T12:29:34.124 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:29:33 smithi079 ceph-mon[23699]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-09-21T12:29:34.124 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:29:33 smithi079 ceph-mon[23699]: from='client.? 172.21.15.79:0/19038619' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-21T12:29:34.124 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:29:33 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:34.124 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:29:33 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:34.124 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:29:33 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:34.124 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:29:33 smithi079 ceph-mon[23699]: pgmap v31: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:29:34.124 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:29:33 smithi079 ceph-mon[23699]: pgmap v32: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:29:34.124 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:29:33 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:34.124 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:29:33 smithi079 ceph-mon[30023]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-09-21T12:29:34.124 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:29:33 smithi079 ceph-mon[30023]: from='client.? 172.21.15.79:0/19038619' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-21T12:29:34.124 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:29:33 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:34.124 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:29:33 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:34.125 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:29:33 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:34.125 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:29:33 smithi079 ceph-mon[30023]: pgmap v31: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:29:34.125 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:29:33 smithi079 ceph-mon[30023]: pgmap v32: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:29:34.125 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:29:33 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:34.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:29:33 smithi104 ceph-mon[25639]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-09-21T12:29:34.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:29:33 smithi104 ceph-mon[25639]: from='client.? 172.21.15.79:0/19038619' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-21T12:29:34.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:29:33 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:34.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:29:33 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:34.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:29:33 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:34.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:29:33 smithi104 ceph-mon[25639]: pgmap v31: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:29:34.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:29:33 smithi104 ceph-mon[25639]: pgmap v32: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:29:34.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:29:33 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:35.734 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182435 got 107374182437 for osd.3 2024-09-21T12:29:35.734 DEBUG:teuthology.parallel:result is None 2024-09-21T12:29:36.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:29:36 smithi079 ceph-mon[23699]: pgmap v33: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:29:36.692 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:29:36 smithi079 ceph-mon[30023]: pgmap v33: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:29:36.836 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:29:36.926 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:29:36 smithi104 ceph-mon[25639]: pgmap v33: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:29:38.925 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:29:38 smithi079 ceph-mon[30023]: pgmap v34: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:29:38.925 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:29:38 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:38.925 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:29:38 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T12:29:38.925 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:29:38 smithi079 ceph-mon[23699]: pgmap v34: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:29:38.926 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:29:38 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:38.926 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:29:38 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T12:29:38.926 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:29:38 smithi104 ceph-mon[25639]: pgmap v34: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:29:38.926 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:29:38 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:38.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:29:38 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T12:29:40.797 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:29:40 smithi079 ceph-mon[30023]: pgmap v35: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:29:40.797 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:29:40 smithi079 ceph-mon[23699]: pgmap v35: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:29:40.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:29:40 smithi104 ceph-mon[25639]: pgmap v35: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:29:41.362 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:29:41.514 INFO:teuthology.orchestra.run.smithi079.stdout:231928234004 2024-09-21T12:29:41.600 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:29:41 smithi079 ceph-mon[23699]: from='client.? 172.21.15.79:0/2154104674' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-21T12:29:41.600 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:29:41 smithi079 ceph-mon[30023]: from='client.? 172.21.15.79:0/2154104674' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-21T12:29:41.601 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Sep 21 12:29:41 smithi079 systemd[1]: ceph-33168022-7814-11ef-baf6-efdc52797490@alertmanager.a.service: Deactivated successfully. 2024-09-21T12:29:41.601 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Sep 21 12:29:41 smithi079 systemd[1]: Stopped Ceph alertmanager.a for 33168022-7814-11ef-baf6-efdc52797490. 2024-09-21T12:29:41.601 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Sep 21 12:29:41 smithi079 systemd[1]: Starting Ceph alertmanager.a for 33168022-7814-11ef-baf6-efdc52797490... 2024-09-21T12:29:41.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:29:41 smithi104 ceph-mon[25639]: from='client.? 172.21.15.79:0/2154104674' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-21T12:29:42.151 INFO:teuthology.orchestra.run.smithi079.stdout:197568495642 2024-09-21T12:29:42.796 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:29:42 smithi079 ceph-mon[30023]: pgmap v36: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:29:42.796 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:29:42 smithi079 ceph-mon[30023]: from='client.? 172.21.15.79:0/390325805' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-21T12:29:42.796 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:29:42 smithi079 ceph-mon[23699]: pgmap v36: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:29:42.797 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:29:42 smithi079 ceph-mon[23699]: from='client.? 172.21.15.79:0/390325805' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-21T12:29:42.926 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:29:42 smithi104 ceph-mon[25639]: pgmap v36: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:29:42.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:29:42 smithi104 ceph-mon[25639]: from='client.? 172.21.15.79:0/390325805' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-21T12:29:43.222 INFO:tasks.cephadm.ceph_manager.ceph:need seq 231928234000 got 231928234004 for osd.7 2024-09-21T12:29:43.223 DEBUG:teuthology.parallel:result is None 2024-09-21T12:29:43.404 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:29:43 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: ::ffff:172.21.15.104 - - [21/Sep/2024:12:29:43] "GET /metrics HTTP/1.1" 200 26277 "" "Prometheus/2.43.0" 2024-09-21T12:29:44.231 INFO:teuthology.orchestra.run.smithi079.stdout:128849018914 2024-09-21T12:29:44.601 INFO:teuthology.orchestra.run.smithi079.stdout:34359738420 2024-09-21T12:29:44.676 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:29:44 smithi104 ceph-mon[25639]: pgmap v37: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:29:44.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:29:44 smithi104 ceph-mon[25639]: from='client.? 172.21.15.79:0/2901339671' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-21T12:29:44.843 INFO:teuthology.orchestra.run.smithi079.stdout:77309411372 2024-09-21T12:29:44.843 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:29:44 smithi079 ceph-mon[23699]: pgmap v37: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:29:44.843 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:29:44 smithi079 ceph-mon[23699]: from='client.? 172.21.15.79:0/2901339671' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-21T12:29:44.843 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:29:44 smithi079 ceph-mon[30023]: pgmap v37: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:29:44.844 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:29:44 smithi079 ceph-mon[30023]: from='client.? 172.21.15.79:0/2901339671' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-21T12:29:45.258 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738415 got 34359738420 for osd.0 2024-09-21T12:29:45.258 DEBUG:teuthology.parallel:result is None 2024-09-21T12:29:45.926 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:29:45 smithi104 ceph-mon[25639]: from='client.? 172.21.15.79:0/2008334365' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-21T12:29:45.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:29:45 smithi104 ceph-mon[25639]: from='client.? 172.21.15.79:0/2987926589' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-21T12:29:45.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:29:45 smithi079 ceph-mon[23699]: from='client.? 172.21.15.79:0/2008334365' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-21T12:29:45.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:29:45 smithi079 ceph-mon[23699]: from='client.? 172.21.15.79:0/2987926589' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-21T12:29:45.942 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:29:45 smithi079 ceph-mon[30023]: from='client.? 172.21.15.79:0/2008334365' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-21T12:29:45.942 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:29:45 smithi079 ceph-mon[30023]: from='client.? 172.21.15.79:0/2987926589' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-21T12:29:46.196 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411369 got 77309411372 for osd.2 2024-09-21T12:29:46.196 DEBUG:teuthology.parallel:result is None 2024-09-21T12:29:46.605 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:29:46 smithi079 ceph-mon[23699]: pgmap v38: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:29:46.605 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:29:46 smithi079 ceph-mon[30023]: pgmap v38: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:29:46.630 INFO:tasks.cephadm.ceph_manager.ceph:need seq 197568495638 got 197568495642 for osd.6 2024-09-21T12:29:46.630 DEBUG:teuthology.parallel:result is None 2024-09-21T12:29:46.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:29:46 smithi104 ceph-mon[25639]: pgmap v38: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:29:47.048 INFO:teuthology.orchestra.run.smithi079.stdout:163208757279 2024-09-21T12:29:47.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:29:47 smithi079 ceph-mon[23699]: from='client.? 172.21.15.79:0/3569769985' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-21T12:29:47.692 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:29:47 smithi079 ceph-mon[30023]: from='client.? 172.21.15.79:0/3569769985' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-21T12:29:47.823 INFO:teuthology.orchestra.run.smithi079.stdout:55834574897 2024-09-21T12:29:47.926 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:29:47 smithi104 ceph-mon[25639]: from='client.? 172.21.15.79:0/3569769985' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-21T12:29:48.090 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Sep 21 12:29:47 smithi079 podman[54474]: 2024-09-21 12:29:47.816763489 +0000 UTC m=+0.094652827 image pull c8568f914cd25b2062c44e9f79f9c18da6e3b85fe0c47a12a2191c61426c2b19 quay.io/prometheus/alertmanager:v0.25.0 2024-09-21T12:29:48.110 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757275 got 163208757279 for osd.5 2024-09-21T12:29:48.111 DEBUG:teuthology.parallel:result is None 2024-09-21T12:29:48.689 INFO:tasks.cephadm.ceph_manager.ceph:need seq 128849018910 got 128849018914 for osd.4 2024-09-21T12:29:48.689 DEBUG:teuthology.parallel:result is None 2024-09-21T12:29:48.837 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:29:48 smithi079 ceph-mon[23699]: pgmap v39: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:29:48.837 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:29:48 smithi079 ceph-mon[23699]: from='client.? 172.21.15.79:0/513623021' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-21T12:29:48.837 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:29:48 smithi079 ceph-mon[30023]: pgmap v39: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:29:48.837 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:29:48 smithi079 ceph-mon[30023]: from='client.? 172.21.15.79:0/513623021' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-21T12:29:48.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:29:48 smithi104 ceph-mon[25639]: pgmap v39: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:29:48.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:29:48 smithi104 ceph-mon[25639]: from='client.? 172.21.15.79:0/513623021' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-21T12:29:49.138 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Sep 21 12:29:48 smithi079 podman[54474]: 2024-09-21 12:29:48.835203116 +0000 UTC m=+1.113092483 volume create fb7afddfd57c7dc4cf35ece272dff3a08e91c7994cbce1073329340ff0f6d750 2024-09-21T12:29:49.139 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Sep 21 12:29:48 smithi079 podman[54474]: 2024-09-21 12:29:48.952633022 +0000 UTC m=+1.230522347 container create 1451b4f42e50f68a2c8a80f740525dfa157c81c4af6d868c35121a760f6b3354 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-33168022-7814-11ef-baf6-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-21T12:29:49.442 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Sep 21 12:29:49 smithi079 podman[54474]: 2024-09-21 12:29:49.230760115 +0000 UTC m=+1.508649452 container init 1451b4f42e50f68a2c8a80f740525dfa157c81c4af6d868c35121a760f6b3354 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-33168022-7814-11ef-baf6-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-21T12:29:49.442 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Sep 21 12:29:49 smithi079 podman[54474]: 2024-09-21 12:29:49.233746188 +0000 UTC m=+1.511635511 container start 1451b4f42e50f68a2c8a80f740525dfa157c81c4af6d868c35121a760f6b3354 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-33168022-7814-11ef-baf6-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-21T12:29:49.442 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Sep 21 12:29:49 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-alertmanager-a[54523]: ts=2024-09-21T12:29:49.247Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-09-21T12:29:49.442 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Sep 21 12:29:49 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-alertmanager-a[54523]: ts=2024-09-21T12:29:49.247Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-09-21T12:29:49.442 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Sep 21 12:29:49 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-alertmanager-a[54523]: ts=2024-09-21T12:29:49.248Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.79 port=9094 2024-09-21T12:29:49.442 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Sep 21 12:29:49 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-alertmanager-a[54523]: ts=2024-09-21T12:29:49.249Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-09-21T12:29:49.442 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Sep 21 12:29:49 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-alertmanager-a[54523]: ts=2024-09-21T12:29:49.272Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-21T12:29:49.442 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Sep 21 12:29:49 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-alertmanager-a[54523]: ts=2024-09-21T12:29:49.272Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-21T12:29:49.442 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Sep 21 12:29:49 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-alertmanager-a[54523]: ts=2024-09-21T12:29:49.275Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-09-21T12:29:49.442 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Sep 21 12:29:49 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-alertmanager-a[54523]: ts=2024-09-21T12:29:49.275Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-09-21T12:29:49.443 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Sep 21 12:29:49 smithi079 bash[54474]: 1451b4f42e50f68a2c8a80f740525dfa157c81c4af6d868c35121a760f6b3354 2024-09-21T12:29:49.443 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Sep 21 12:29:49 smithi079 systemd[1]: Started Ceph alertmanager.a for 33168022-7814-11ef-baf6-efdc52797490. 2024-09-21T12:29:49.989 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574893 got 55834574897 for osd.1 2024-09-21T12:29:49.989 DEBUG:teuthology.parallel:result is None 2024-09-21T12:29:49.989 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-09-21T12:29:49.990 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 33168022-7814-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T12:29:50.492 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:29:50.607 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:29:50 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:50.607 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:29:50 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:50.608 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:29:50 smithi079 ceph-mon[30023]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-21T12:29:50.608 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:29:50 smithi079 ceph-mon[30023]: pgmap v40: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:29:50.608 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:29:50 smithi079 ceph-mon[30023]: Reconfiguring daemon prometheus.a on smithi104 2024-09-21T12:29:50.608 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:29:50 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:50.608 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:29:50 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:50.608 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:29:50 smithi079 ceph-mon[23699]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-21T12:29:50.608 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:29:50 smithi079 ceph-mon[23699]: pgmap v40: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:29:50.608 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:29:50 smithi079 ceph-mon[23699]: Reconfiguring daemon prometheus.a on smithi104 2024-09-21T12:29:50.620 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:29:50 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:50.620 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:29:50 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:50.620 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:29:50 smithi104 ceph-mon[25639]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-21T12:29:50.621 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:29:50 smithi104 ceph-mon[25639]: pgmap v40: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:29:50.621 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:29:50 smithi104 ceph-mon[25639]: Reconfiguring daemon prometheus.a on smithi104 2024-09-21T12:29:51.573 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Sep 21 12:29:51 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-alertmanager-a[54523]: ts=2024-09-21T12:29:51.250Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.00087416s 2024-09-21T12:29:52.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:29:52 smithi079 ceph-mon[23699]: pgmap v41: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:29:52.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:29:52 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:52.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:29:52 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:52.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:29:52 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:52.693 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:29:52 smithi079 ceph-mon[30023]: pgmap v41: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:29:52.693 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:29:52 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:52.693 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:29:52 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:52.693 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:29:52 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:52.927 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:29:52 smithi104 systemd[1]: Stopping Ceph prometheus.a for 33168022-7814-11ef-baf6-efdc52797490... 2024-09-21T12:29:52.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:29:52 smithi104 ceph-mon[25639]: pgmap v41: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:29:52.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:29:52 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:52.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:29:52 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:52.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:29:52 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:53.265 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:29:52 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[44421]: ts=2024-09-21T12:29:52.971Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-09-21T12:29:53.265 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:29:52 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[44421]: ts=2024-09-21T12:29:52.971Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-09-21T12:29:53.265 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:29:52 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[44421]: ts=2024-09-21T12:29:52.972Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-09-21T12:29:53.266 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:29:52 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[44421]: ts=2024-09-21T12:29:52.972Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-09-21T12:29:53.266 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:29:52 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[44421]: ts=2024-09-21T12:29:52.972Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-09-21T12:29:53.266 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:29:52 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[44421]: ts=2024-09-21T12:29:52.972Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-09-21T12:29:53.266 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:29:52 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[44421]: ts=2024-09-21T12:29:52.972Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-09-21T12:29:53.266 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:29:52 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[44421]: ts=2024-09-21T12:29:52.972Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-09-21T12:29:53.266 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:29:52 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[44421]: ts=2024-09-21T12:29:52.972Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-09-21T12:29:53.266 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:29:52 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[44421]: ts=2024-09-21T12:29:52.972Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-09-21T12:29:53.266 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:29:52 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[44421]: ts=2024-09-21T12:29:52.972Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-09-21T12:29:53.266 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:29:52 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[44421]: ts=2024-09-21T12:29:52.972Z caller=main.go:1147 level=info msg="See you next time!" 2024-09-21T12:29:53.266 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:29:52 smithi104 podman[49038]: 2024-09-21 12:29:52.982004085 +0000 UTC m=+0.144148366 container died fee7afdd333030590a42ed3f9466a294be0acd80e04bdbdd0dd5c2a7f9e39b42 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T12:29:53.536 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:29:53 smithi104 podman[49038]: 2024-09-21 12:29:53.363129771 +0000 UTC m=+0.525274041 container cleanup fee7afdd333030590a42ed3f9466a294be0acd80e04bdbdd0dd5c2a7f9e39b42 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T12:29:53.536 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:29:53 smithi104 bash[49038]: ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a 2024-09-21T12:29:53.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:29:53 smithi079 ceph-mon[23699]: pgmap v42: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:29:53.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:29:53 smithi079 ceph-mon[23699]: pgmap v43: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:29:53.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:29:53 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:53.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:29:53 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:53.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:29:53 smithi079 ceph-mon[23699]: pgmap v44: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:29:53.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:29:53 smithi079 ceph-mon[23699]: pgmap v45: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:29:53.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:29:53 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:53.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:29:53 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T12:29:53.693 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:29:53 smithi079 ceph-mon[30023]: pgmap v42: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:29:53.693 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:29:53 smithi079 ceph-mon[30023]: pgmap v43: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:29:53.693 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:29:53 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:53.693 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:29:53 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:53.693 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:29:53 smithi079 ceph-mon[30023]: pgmap v44: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:29:53.693 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:29:53 smithi079 ceph-mon[30023]: pgmap v45: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:29:53.693 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:29:53 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:53.693 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:29:53 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T12:29:53.789 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:29:53 smithi104 podman[49050]: 2024-09-21 12:29:53.639725356 +0000 UTC m=+0.660226719 container remove fee7afdd333030590a42ed3f9466a294be0acd80e04bdbdd0dd5c2a7f9e39b42 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T12:29:53.790 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:29:53 smithi104 ceph-mon[25639]: pgmap v42: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:29:53.790 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:29:53 smithi104 ceph-mon[25639]: pgmap v43: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:29:53.790 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:29:53 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:53.790 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:29:53 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:53.790 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:29:53 smithi104 ceph-mon[25639]: pgmap v44: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:29:53.790 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:29:53 smithi104 ceph-mon[25639]: pgmap v45: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:29:53.790 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:29:53 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:53.790 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:29:53 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T12:29:54.132 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:29:53 smithi104 systemd[1]: ceph-33168022-7814-11ef-baf6-efdc52797490@prometheus.a.service: Deactivated successfully. 2024-09-21T12:29:54.133 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:29:53 smithi104 systemd[1]: Stopped Ceph prometheus.a for 33168022-7814-11ef-baf6-efdc52797490. 2024-09-21T12:29:54.133 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:29:53 smithi104 systemd[1]: ceph-33168022-7814-11ef-baf6-efdc52797490@prometheus.a.service: Consumed 1.118s CPU time. 2024-09-21T12:29:54.133 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:29:53 smithi104 systemd[1]: Starting Ceph prometheus.a for 33168022-7814-11ef-baf6-efdc52797490... 2024-09-21T12:29:54.133 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:29:53 smithi104 podman[49166]: 2024-09-21 12:29:53.901406507 +0000 UTC m=+0.017255417 image pull a07b618ecd1dce142bce4c52f0e80982eaf1f14265a2415c2d35978ccaa0a464 quay.io/prometheus/prometheus:v2.43.0 2024-09-21T12:29:54.133 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:29:54 smithi104 podman[49166]: 2024-09-21 12:29:54.024230125 +0000 UTC m=+0.140079047 container create b307869246eeb9deb64c060fa0aa22ece21a2a71d0b274636b9778826bb98ecb (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T12:29:54.427 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:29:54 smithi104 podman[49166]: 2024-09-21 12:29:54.234469326 +0000 UTC m=+0.350318209 container init b307869246eeb9deb64c060fa0aa22ece21a2a71d0b274636b9778826bb98ecb (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T12:29:54.427 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:29:54 smithi104 podman[49166]: 2024-09-21 12:29:54.237178251 +0000 UTC m=+0.353027131 container start b307869246eeb9deb64c060fa0aa22ece21a2a71d0b274636b9778826bb98ecb (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T12:29:54.427 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:29:54 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[49181]: ts=2024-09-21T12:29:54.260Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-09-21T12:29:54.427 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:29:54 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[49181]: ts=2024-09-21T12:29:54.260Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-09-21T12:29:54.428 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:29:54 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[49181]: ts=2024-09-21T12:29:54.260Z caller=main.go:570 level=info host_details="(Linux 5.14.0-508.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024 x86_64 smithi104 (none))" 2024-09-21T12:29:54.428 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:29:54 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[49181]: ts=2024-09-21T12:29:54.260Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-09-21T12:29:54.428 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:29:54 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[49181]: ts=2024-09-21T12:29:54.260Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-09-21T12:29:54.428 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:29:54 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[49181]: ts=2024-09-21T12:29:54.261Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-09-21T12:29:54.428 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:29:54 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[49181]: ts=2024-09-21T12:29:54.262Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-09-21T12:29:54.428 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:29:54 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[49181]: ts=2024-09-21T12:29:54.263Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-09-21T12:29:54.428 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:29:54 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[49181]: ts=2024-09-21T12:29:54.263Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-09-21T12:29:54.428 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:29:54 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[49181]: ts=2024-09-21T12:29:54.265Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-09-21T12:29:54.428 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:29:54 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[49181]: ts=2024-09-21T12:29:54.265Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=2.218µs 2024-09-21T12:29:54.428 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:29:54 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[49181]: ts=2024-09-21T12:29:54.265Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-09-21T12:29:54.428 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:29:54 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[49181]: ts=2024-09-21T12:29:54.267Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=1 2024-09-21T12:29:54.428 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:29:54 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[49181]: ts=2024-09-21T12:29:54.267Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=1 2024-09-21T12:29:54.428 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:29:54 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[49181]: ts=2024-09-21T12:29:54.267Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=26.875µs wal_replay_duration=1.511106ms wbl_replay_duration=108ns total_replay_duration=1.562518ms 2024-09-21T12:29:54.428 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:29:54 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[49181]: ts=2024-09-21T12:29:54.267Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-09-21T12:29:54.429 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:29:54 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[49181]: ts=2024-09-21T12:29:54.267Z caller=main.go:1029 level=info msg="TSDB started" 2024-09-21T12:29:54.429 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:29:54 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[49181]: ts=2024-09-21T12:29:54.267Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-09-21T12:29:54.429 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:29:54 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[49181]: ts=2024-09-21T12:29:54.288Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=20.23186ms db_storage=922ns remote_storage=1.051µs web_handler=354ns query_engine=550ns scrape=428.494µs scrape_sd=78.502µs notify=9.364µs notify_sd=8.194µs rules=19.40758ms tracing=5.38µs 2024-09-21T12:29:54.429 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:29:54 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[49181]: ts=2024-09-21T12:29:54.288Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-09-21T12:29:54.429 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:29:54 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[49181]: ts=2024-09-21T12:29:54.288Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-09-21T12:29:54.429 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:29:54 smithi104 bash[49166]: b307869246eeb9deb64c060fa0aa22ece21a2a71d0b274636b9778826bb98ecb 2024-09-21T12:29:54.429 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:29:54 smithi104 systemd[1]: Started Ceph prometheus.a for 33168022-7814-11ef-baf6-efdc52797490. 2024-09-21T12:29:54.684 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:29:54 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: [21/Sep/2024:12:29:54] ENGINE Bus STOPPING 2024-09-21T12:29:54.684 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:29:54 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: [21/Sep/2024:12:29:54] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-21T12:29:54.684 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:29:54 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: [21/Sep/2024:12:29:54] ENGINE Bus STOPPED 2024-09-21T12:29:54.684 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:29:54 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: [21/Sep/2024:12:29:54] ENGINE Bus STARTING 2024-09-21T12:29:54.684 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:29:54 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: [21/Sep/2024:12:29:54] ENGINE Serving on http://:::9283 2024-09-21T12:29:54.684 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:29:54 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: [21/Sep/2024:12:29:54] ENGINE Bus STARTED 2024-09-21T12:29:54.684 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:29:54 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: [21/Sep/2024:12:29:54] ENGINE Bus STOPPING 2024-09-21T12:29:54.685 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:29:54 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:54.685 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:29:54 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:54.685 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:29:54 smithi079 ceph-mon[23699]: pgmap v46: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:29:54.685 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:29:54 smithi079 ceph-mon[23699]: pgmap v47: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:29:54.685 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:29:54 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:54.685 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:29:54 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:54.685 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:29:54 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:54.685 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:29:54 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-21T12:29:54.685 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:29:54 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi079.front.sepia.ceph.com:9093"}]: dispatch 2024-09-21T12:29:54.685 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:29:54 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:54.686 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:29:54 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-21T12:29:54.686 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:29:54 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi104.front.sepia.ceph.com:9095"}]: dispatch 2024-09-21T12:29:54.686 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:29:54 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:54.686 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:29:54 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-21T12:29:54.686 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:29:54 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi104.front.sepia.ceph.com:3000"}]: dispatch 2024-09-21T12:29:54.686 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:29:54 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:54.686 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:29:54 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:29:54.686 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:29:54 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:29:54.686 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:29:54 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:29:54.686 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:29:54 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:54.686 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:29:54 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:54.686 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:29:54 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:54.687 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:29:54 smithi079 ceph-mon[30023]: pgmap v46: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:29:54.687 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:29:54 smithi079 ceph-mon[30023]: pgmap v47: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:29:54.687 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:29:54 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:54.687 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:29:54 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:54.687 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:29:54 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:54.687 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:29:54 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-21T12:29:54.687 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:29:54 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi079.front.sepia.ceph.com:9093"}]: dispatch 2024-09-21T12:29:54.687 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:29:54 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:54.687 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:29:54 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-21T12:29:54.687 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:29:54 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi104.front.sepia.ceph.com:9095"}]: dispatch 2024-09-21T12:29:54.687 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:29:54 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:54.687 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:29:54 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-21T12:29:54.687 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:29:54 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi104.front.sepia.ceph.com:3000"}]: dispatch 2024-09-21T12:29:54.687 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:29:54 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:54.688 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:29:54 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:29:54.688 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:29:54 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:29:54.688 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:29:54 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:29:54.688 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:29:54 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:54.708 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-21T12:29:54.709 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-09-21T12:29:54.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:29:54 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:54.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:29:54 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:54.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:29:54 smithi104 ceph-mon[25639]: pgmap v46: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:29:54.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:29:54 smithi104 ceph-mon[25639]: pgmap v47: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:29:54.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:29:54 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:54.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:29:54 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:54.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:29:54 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:54.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:29:54 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-21T12:29:54.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:29:54 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi079.front.sepia.ceph.com:9093"}]: dispatch 2024-09-21T12:29:54.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:29:54 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:54.928 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:29:54 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-21T12:29:54.928 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:29:54 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi104.front.sepia.ceph.com:9095"}]: dispatch 2024-09-21T12:29:54.928 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:29:54 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:54.928 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:29:54 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-21T12:29:54.928 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:29:54 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi104.front.sepia.ceph.com:3000"}]: dispatch 2024-09-21T12:29:54.928 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:29:54 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:54.928 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:29:54 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:29:54.928 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:29:54 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:29:54.928 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:29:54 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:29:54.928 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:29:54 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:55.442 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:29:55 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: [21/Sep/2024:12:29:55] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-21T12:29:55.442 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:29:55 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: [21/Sep/2024:12:29:55] ENGINE Bus STOPPED 2024-09-21T12:29:55.442 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:29:55 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: [21/Sep/2024:12:29:55] ENGINE Bus STARTING 2024-09-21T12:29:55.442 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:29:55 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: [21/Sep/2024:12:29:55] ENGINE Serving on http://:::9283 2024-09-21T12:29:55.442 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:29:55 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: [21/Sep/2024:12:29:55] ENGINE Bus STARTED 2024-09-21T12:29:55.442 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:29:55 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: [21/Sep/2024:12:29:55] ENGINE Bus STOPPING 2024-09-21T12:29:55.797 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:29:55 smithi079 ceph-mon[23699]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-21T12:29:55.797 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:29:55 smithi079 ceph-mon[23699]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi079.front.sepia.ceph.com:9093"}]: dispatch 2024-09-21T12:29:55.798 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:29:55 smithi079 ceph-mon[23699]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-21T12:29:55.798 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:29:55 smithi079 ceph-mon[23699]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi104.front.sepia.ceph.com:9095"}]: dispatch 2024-09-21T12:29:55.798 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:29:55 smithi079 ceph-mon[23699]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-21T12:29:55.798 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:29:55 smithi079 ceph-mon[23699]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi104.front.sepia.ceph.com:3000"}]: dispatch 2024-09-21T12:29:55.798 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:29:55 smithi079 ceph-mon[23699]: pgmap v48: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:29:55.798 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:29:55 smithi079 ceph-mon[23699]: pgmap v49: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:29:55.798 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:29:55 smithi079 ceph-mon[23699]: from='client.24503 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T12:29:55.798 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:29:55 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: [21/Sep/2024:12:29:55] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-21T12:29:55.798 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:29:55 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: [21/Sep/2024:12:29:55] ENGINE Bus STOPPED 2024-09-21T12:29:55.798 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:29:55 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: [21/Sep/2024:12:29:55] ENGINE Bus STARTING 2024-09-21T12:29:55.798 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:29:55 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: [21/Sep/2024:12:29:55] ENGINE Serving on http://:::9283 2024-09-21T12:29:55.799 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:29:55 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: [21/Sep/2024:12:29:55] ENGINE Bus STARTED 2024-09-21T12:29:55.799 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:29:55 smithi079 ceph-mon[30023]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-21T12:29:55.799 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:29:55 smithi079 ceph-mon[30023]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi079.front.sepia.ceph.com:9093"}]: dispatch 2024-09-21T12:29:55.799 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:29:55 smithi079 ceph-mon[30023]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-21T12:29:55.799 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:29:55 smithi079 ceph-mon[30023]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi104.front.sepia.ceph.com:9095"}]: dispatch 2024-09-21T12:29:55.799 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:29:55 smithi079 ceph-mon[30023]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-21T12:29:55.799 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:29:55 smithi079 ceph-mon[30023]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi104.front.sepia.ceph.com:3000"}]: dispatch 2024-09-21T12:29:55.799 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:29:55 smithi079 ceph-mon[30023]: pgmap v48: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:29:55.799 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:29:55 smithi079 ceph-mon[30023]: pgmap v49: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:29:55.799 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:29:55 smithi079 ceph-mon[30023]: from='client.24503 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T12:29:55.926 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:29:55 smithi104 ceph-mon[25639]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-21T12:29:55.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:29:55 smithi104 ceph-mon[25639]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi079.front.sepia.ceph.com:9093"}]: dispatch 2024-09-21T12:29:55.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:29:55 smithi104 ceph-mon[25639]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-21T12:29:55.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:29:55 smithi104 ceph-mon[25639]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi104.front.sepia.ceph.com:9095"}]: dispatch 2024-09-21T12:29:55.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:29:55 smithi104 ceph-mon[25639]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-21T12:29:55.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:29:55 smithi104 ceph-mon[25639]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi104.front.sepia.ceph.com:3000"}]: dispatch 2024-09-21T12:29:55.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:29:55 smithi104 ceph-mon[25639]: pgmap v48: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:29:55.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:29:55 smithi104 ceph-mon[25639]: pgmap v49: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:29:55.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:29:55 smithi104 ceph-mon[25639]: from='client.24503 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T12:29:55.936 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":49,"stamp":"2024-09-21T12:29:54.436171+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":3,"kb":749961216,"kb_used":226544,"kb_used_data":6292,"kb_used_omap":12,"kb_used_meta":220147,"kb_avail":749734672,"statfs":{"total":767960285184,"available":767728304128,"internally_reserved":0,"allocated":6443008,"data_stored":3677856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":225431097},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.349688"},"pg_stats":[{"pgid":"1.0","version":"59'128","reported_seq":255,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-21T12:28:40.392802+0000","last_change":"2024-09-21T12:28:13.865028+0000","last_active":"2024-09-21T12:28:40.392802+0000","last_peered":"2024-09-21T12:28:40.392802+0000","last_clean":"2024-09-21T12:28:40.392802+0000","last_became_active":"2024-09-21T12:28:13.864686+0000","last_became_peered":"2024-09-21T12:28:13.864686+0000","last_unstale":"2024-09-21T12:28:40.392802+0000","last_undegraded":"2024-09-21T12:28:40.392802+0000","last_fullsized":"2024-09-21T12:28:40.392802+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:19.398242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:26:19.398242+0000","last_clean_scrub_stamp":"2024-09-21T12:26:19.398242+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:23:11.581508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234007,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28172,"kb_used_data":1152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716980,"statfs":{"total":95995035648,"available":95966187520,"internally_reserved":0,"allocated":1179648,"data_stored":828925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 12:29:13 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.63400000000000001}]},{"osd":1,"last update":"Sat Sep 21 12:29:13 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"last":0.58699999999999997}]},{"osd":2,"last update":"Sat Sep 21 12:29:13 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.627},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.0109999999999999,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.57199999999999995}]},{"osd":3,"last update":"Sat Sep 21 12:29:13 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.0629999999999999,"5min":1.0629999999999999,"15min":1.0629999999999999},"last":0.59699999999999998}]},{"osd":4,"last update":"Sat Sep 21 12:29:13 2024","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"last":0.443},{"interface":"front","average":{"1min":0.443,"5min":0.443,"15min":0.443},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.49299999999999999}]},{"osd":5,"last update":"Sat Sep 21 12:29:13 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":1.0429999999999999,"5min":1.0429999999999999,"15min":1.0429999999999999},"last":0.47899999999999998}]},{"osd":6,"last update":"Sat Sep 21 12:29:13 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":1.083,"5min":1.083,"15min":1.083},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.45800000000000002}]}]},{"osd":6,"up_from":46,"seq":197568495645,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28156,"kb_used_data":1140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716996,"statfs":{"total":95995035648,"available":95966203904,"internally_reserved":0,"allocated":1167360,"data_stored":828098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 12:29:52 2024","interfaces":[{"interface":"back","average":{"1min":1.623,"5min":0.71799999999999997,"15min":0.56699999999999995},"min":{"1min":0.35099999999999998,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":19.552,"5min":19.552,"15min":19.552},"last":0.50600000000000001},{"interface":"front","average":{"1min":1.617,"5min":0.749,"15min":0.60399999999999998},"min":{"1min":0.38700000000000001,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":19.523,"5min":19.523,"15min":19.523},"last":0.56799999999999995}]},{"osd":1,"last update":"Sat Sep 21 12:29:52 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.51400000000000001,"15min":0.51600000000000001},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.60999999999999999,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.52500000000000002,"15min":0.52700000000000002},"min":{"1min":0.40899999999999997,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.66200000000000003,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.42499999999999999}]},{"osd":2,"last update":"Sat Sep 21 12:29:52 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.52500000000000002,"15min":0.53100000000000003},"min":{"1min":0.39500000000000002,"5min":0.378,"15min":0.378},"max":{"1min":0.622,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.503,"15min":0.5},"min":{"1min":0.39700000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.58799999999999997}]},{"osd":3,"last update":"Sat Sep 21 12:29:52 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.54100000000000004,"15min":0.54500000000000004},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.64000000000000001,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.53500000000000003}]},{"osd":4,"last update":"Sat Sep 21 12:29:52 2024","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.68300000000000005,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.42499999999999999,"15min":0.41899999999999998},"min":{"1min":0.29999999999999999,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.72599999999999998,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.39900000000000002}]},{"osd":5,"last update":"Sat Sep 21 12:29:52 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.44600000000000001,"15min":0.442},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.71299999999999997,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.443,"15min":0.437},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.67500000000000004,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.44800000000000001}]},{"osd":7,"last update":"Sat Sep 21 12:29:11 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.496,"15min":0.496},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.63},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.64200000000000002}]}]},{"osd":5,"up_from":38,"seq":163208757281,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27592,"kb_used_data":572,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717560,"statfs":{"total":95995035648,"available":95966781440,"internally_reserved":0,"allocated":585728,"data_stored":238557,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 12:29:25 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.50700000000000001,"15min":0.51300000000000001},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.57799999999999996,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.51300000000000001,"15min":0.52100000000000002},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.61699999999999999,"5min":0.69199999999999995,"15min":0.69199999999999995},"last":0.55700000000000005}]},{"osd":1,"last update":"Sat Sep 21 12:29:25 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.53100000000000003,"15min":0.53900000000000003},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.66300000000000003,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.505,"15min":0.51100000000000001},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.54700000000000004,"5min":0.753,"15min":0.753},"last":0.49199999999999999}]},{"osd":2,"last update":"Sat Sep 21 12:29:25 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.51400000000000001,"15min":0.52000000000000002},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.66500000000000004,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.502},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.53600000000000003,"15min":0.54500000000000004},"min":{"1min":0.35699999999999998,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.63100000000000001,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.52400000000000002}]},{"osd":3,"last update":"Sat Sep 21 12:29:25 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.52200000000000002,"15min":0.52800000000000002},"min":{"1min":0.376,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.64400000000000002,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.504,"5min":0.53400000000000003,"15min":0.53900000000000003},"min":{"1min":0.38400000000000001,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.63600000000000001,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.54900000000000004}]},{"osd":4,"last update":"Sat Sep 21 12:29:25 2024","interfaces":[{"interface":"back","average":{"1min":0.39000000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"min":{"1min":0.28899999999999998,"5min":0.215,"15min":0.215},"max":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.39800000000000002,"15min":0.39600000000000002},"min":{"1min":0.28399999999999997,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.61099999999999999,"5min":0.753,"15min":0.753},"last":0.30399999999999999}]},{"osd":6,"last update":"Sat Sep 21 12:29:49 2024","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.39700000000000002,"15min":0.39100000000000001},"min":{"1min":0.30599999999999999,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.31900000000000001},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.41299999999999998,"15min":0.40500000000000003},"min":{"1min":0.28699999999999998,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.35799999999999998}]},{"osd":7,"last update":"Sat Sep 21 12:29:11 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.377}]}]},{"osd":4,"up_from":30,"seq":128849018917,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27584,"kb_used_data":564,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717568,"statfs":{"total":95995035648,"available":95966789632,"internally_reserved":0,"allocated":577536,"data_stored":237953,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 12:29:04 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.505,"15min":0.51100000000000001},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.61299999999999999,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.498,"15min":0.50600000000000001},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.60099999999999998,"5min":0.66600000000000004,"15min":0.66600000000000004},"last":0.58099999999999996}]},{"osd":1,"last update":"Sat Sep 21 12:29:04 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.47799999999999998,"15min":0.48199999999999998},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.64900000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.51400000000000001,"15min":0.51600000000000001},"min":{"1min":0.41399999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"last":0.45600000000000002}]},{"osd":2,"last update":"Sat Sep 21 12:29:04 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.51400000000000001,"15min":0.51700000000000002},"min":{"1min":0.38300000000000001,"5min":0.376,"15min":0.376},"max":{"1min":0.66300000000000003,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.46800000000000003,"5min":0.51300000000000001,"15min":0.52100000000000002},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.66800000000000004,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.48999999999999999}]},{"osd":3,"last update":"Sat Sep 21 12:29:04 2024","interfaces":[{"interface":"back","average":{"1min":0.442,"5min":0.50700000000000001,"15min":0.51800000000000002},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.69699999999999995,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.51200000000000001,"15min":0.51300000000000001},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.69499999999999995,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.56599999999999995}]},{"osd":5,"last update":"Sat Sep 21 12:29:26 2024","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.35399999999999998,"15min":0.33800000000000002},"min":{"1min":0.28999999999999998,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.39500000000000002,"15min":0.38300000000000001},"min":{"1min":0.33300000000000002,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.60399999999999998,"5min":0.66400000000000003,"15min":0.66400000000000003},"last":0.53700000000000003}]},{"osd":6,"last update":"Sat Sep 21 12:29:52 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.39300000000000002,"15min":0.38500000000000001},"min":{"1min":0.34799999999999998,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.54500000000000004,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.496,"5min":0.44,"15min":0.43099999999999999},"min":{"1min":0.36499999999999999,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.629,"5min":0.67300000000000004,"15min":0.67300000000000004},"last":0.59399999999999997}]},{"osd":7,"last update":"Sat Sep 21 12:29:13 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.60399999999999998}]}]},{"osd":3,"up_from":25,"seq":107374182442,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27592,"kb_used_data":572,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717560,"statfs":{"total":95995035648,"available":95966781440,"internally_reserved":0,"allocated":585728,"data_stored":238557,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 12:29:47 2024","interfaces":[{"interface":"back","average":{"1min":0.375,"5min":0.38100000000000001,"15min":0.379},"min":{"1min":0.247,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.51700000000000002,"5min":0.68400000000000005,"15min":0.68400000000000005},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.38300000000000001,"5min":0.40100000000000002,"15min":0.40400000000000003},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.53000000000000003,"5min":0.57699999999999996,"15min":0.57699999999999996},"last":0.34999999999999998}]},{"osd":1,"last update":"Sat Sep 21 12:29:47 2024","interfaces":[{"interface":"back","average":{"1min":0.38200000000000001,"5min":0.38800000000000001,"15min":0.38600000000000001},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"last":0.314},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.42599999999999999,"15min":0.42599999999999999},"min":{"1min":0.27400000000000002,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.59299999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"last":0.29799999999999999}]},{"osd":2,"last update":"Sat Sep 21 12:29:47 2024","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.442,"15min":0.442},"min":{"1min":0.29299999999999998,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.56200000000000006,"5min":0.624,"15min":0.624},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.42099999999999999,"15min":0.42599999999999999},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.55600000000000005,"5min":0.63900000000000001,"15min":0.63900000000000001},"last":0.26200000000000001}]},{"osd":4,"last update":"Sat Sep 21 12:29:04 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.57799999999999996,"15min":0.57599999999999996},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.497},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.59299999999999997,"15min":0.59199999999999997},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.94899999999999995,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.42099999999999999}]},{"osd":5,"last update":"Sat Sep 21 12:29:30 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.621,"15min":0.63},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.56999999999999995,"15min":0.56899999999999995},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.438}]},{"osd":6,"last update":"Sat Sep 21 12:29:47 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.60899999999999999,"15min":0.60899999999999999},"min":{"1min":0.46500000000000002,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.998,"5min":0.998,"15min":0.998},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.57499999999999996,"15min":0.57299999999999995},"min":{"1min":0.36799999999999999,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.78400000000000003,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.44900000000000001}]},{"osd":7,"last update":"Sat Sep 21 12:29:12 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.48099999999999998}]}]},{"osd":2,"up_from":18,"seq":77309411375,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27592,"kb_used_data":572,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717560,"statfs":{"total":95995035648,"available":95966781440,"internally_reserved":0,"allocated":585728,"data_stored":238557,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 12:29:23 2024","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.35399999999999998,"15min":0.34599999999999997},"min":{"1min":0.23100000000000001,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.35299999999999998,"15min":0.34499999999999997},"min":{"1min":0.23899999999999999,"5min":0.214,"15min":0.214},"max":{"1min":0.54400000000000004,"5min":0.58099999999999996,"15min":0.58099999999999996},"last":0.34799999999999998}]},{"osd":1,"last update":"Sat Sep 21 12:29:23 2024","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.34899999999999998,"15min":0.34000000000000002},"min":{"1min":0.25800000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.503,"5min":0.53800000000000003,"15min":0.53800000000000003},"last":0.373},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.38200000000000001,"15min":0.376},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.60999999999999999,"5min":0.64200000000000002,"15min":0.64200000000000002},"last":0.38600000000000001}]},{"osd":3,"last update":"Sat Sep 21 12:29:43 2024","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.42799999999999999,"15min":0.42999999999999999},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.57299999999999995,"5min":0.59099999999999997,"15min":0.59099999999999997},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.41199999999999998,"15min":0.41999999999999998},"min":{"1min":0.27000000000000002,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.51900000000000002,"5min":0.62,"15min":0.62},"last":0.30399999999999999}]},{"osd":4,"last update":"Sat Sep 21 12:29:00 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.54000000000000004,"15min":0.53800000000000003},"min":{"1min":0.35899999999999999,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.56699999999999995,"15min":0.56799999999999995},"min":{"1min":0.38300000000000001,"5min":0.379,"15min":0.379},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.53900000000000003}]},{"osd":5,"last update":"Sat Sep 21 12:29:25 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.54700000000000004,"15min":0.55400000000000005},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.53200000000000003,"15min":0.53000000000000003},"min":{"1min":0.38900000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.51100000000000001}]},{"osd":6,"last update":"Sat Sep 21 12:29:48 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.55100000000000005,"15min":0.55500000000000005},"min":{"1min":0.39800000000000002,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.70999999999999996,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.442},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.57199999999999995,"15min":0.57399999999999995},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.81899999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.48699999999999999}]},{"osd":7,"last update":"Sat Sep 21 12:29:10 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.57799999999999996}]}]},{"osd":1,"up_from":13,"seq":55834574899,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27592,"kb_used_data":572,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717560,"statfs":{"total":95995035648,"available":95966781440,"internally_reserved":0,"allocated":585728,"data_stored":238557,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 12:29:01 2024","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.33000000000000002,"15min":0.314},"min":{"1min":0.19600000000000001,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.53100000000000003,"5min":0.63300000000000001,"15min":0.63300000000000001},"last":0.28100000000000003},{"interface":"front","average":{"1min":0.32900000000000001,"5min":0.32400000000000001,"15min":0.309},"min":{"1min":0.21099999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.46700000000000003,"5min":0.56899999999999995,"15min":0.56899999999999995},"last":0.17299999999999999}]},{"osd":2,"last update":"Sat Sep 21 12:29:21 2024","interfaces":[{"interface":"back","average":{"1min":0.37,"5min":0.36199999999999999,"15min":0.35099999999999998},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.54800000000000004,"5min":0.60299999999999998,"15min":0.60299999999999998},"last":0.251},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.36399999999999999,"15min":0.35799999999999998},"min":{"1min":0.252,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.55900000000000005,"5min":0.58599999999999997,"15min":0.58599999999999997},"last":0.27200000000000002}]},{"osd":3,"last update":"Sat Sep 21 12:29:45 2024","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":0.40699999999999997,"15min":0.40699999999999997},"min":{"1min":0.29699999999999999,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.58199999999999996,"5min":0.66800000000000004,"15min":0.66800000000000004},"last":0.29299999999999998},{"interface":"front","average":{"1min":0.42499999999999999,"5min":0.41599999999999998,"15min":0.40799999999999997},"min":{"1min":0.30099999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.31}]},{"osd":4,"last update":"Sat Sep 21 12:29:01 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.55000000000000004,"15min":0.55200000000000005},"min":{"1min":0.374,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.73699999999999999,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.54200000000000004,"15min":0.54300000000000004},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.36599999999999999}]},{"osd":5,"last update":"Sat Sep 21 12:29:26 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.54800000000000004,"15min":0.54500000000000004},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.55800000000000005,"15min":0.55900000000000005},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.747,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.39000000000000001}]},{"osd":6,"last update":"Sat Sep 21 12:29:48 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.57699999999999996,"15min":0.57599999999999996},"min":{"1min":0.40200000000000002,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.57199999999999995,"15min":0.56799999999999995},"min":{"1min":0.39400000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.35799999999999998}]},{"osd":7,"last update":"Sat Sep 21 12:29:11 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.45300000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738423,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32264,"kb_used_data":1148,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93712888,"statfs":{"total":95995035648,"available":95961997312,"internally_reserved":0,"allocated":1175552,"data_stored":828652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 21 12:29:01 2024","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.33100000000000002,"15min":0.313},"min":{"1min":0.216,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":0.497,"5min":0.55800000000000005,"15min":0.55800000000000005},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.34000000000000002,"5min":0.33900000000000002,"15min":0.32900000000000001},"min":{"1min":0.24399999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.53700000000000003,"5min":0.58999999999999997,"15min":0.58999999999999997},"last":0.317}]},{"osd":2,"last update":"Sat Sep 21 12:29:23 2024","interfaces":[{"interface":"back","average":{"1min":0.374,"5min":0.36299999999999999,"15min":0.35499999999999998},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.34200000000000003,"5min":0.35699999999999998,"15min":0.35199999999999998},"min":{"1min":0.224,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.67700000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.33800000000000002}]},{"osd":3,"last update":"Sat Sep 21 12:29:46 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.40999999999999998,"15min":0.40400000000000003},"min":{"1min":0.29699999999999999,"5min":0.252,"15min":0.252},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.42099999999999999,"5min":0.40699999999999997,"15min":0.40000000000000002},"min":{"1min":0.249,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.36899999999999999}]},{"osd":4,"last update":"Sat Sep 21 12:29:01 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.55500000000000005,"15min":0.55300000000000005},"min":{"1min":0.40699999999999997,"5min":0.316,"15min":0.316},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.437},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56899999999999995,"15min":0.56999999999999995},"min":{"1min":0.42799999999999999,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.442}]},{"osd":5,"last update":"Sat Sep 21 12:29:24 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.58599999999999997,"15min":0.59199999999999997},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.54400000000000004,"15min":0.54600000000000004},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.022,"5min":1.022,"15min":1.022},"last":0.45100000000000001}]},{"osd":6,"last update":"Sat Sep 21 12:29:46 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.57199999999999995,"15min":0.57599999999999996},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.995,"5min":0.995,"15min":0.995},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.62,"5min":0.58899999999999997,"15min":0.58399999999999996},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.127,"5min":1.127,"15min":1.127},"last":0.46800000000000003}]},{"osd":7,"last update":"Sat Sep 21 12:29:14 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.95699999999999996,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.42299999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T12:29:55.937 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 33168022-7814-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T12:29:56.151 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:29:56.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:29:56 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:56.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:29:56 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:56.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:29:56 smithi079 ceph-mon[23699]: pgmap v50: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:29:56.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:29:56 smithi079 ceph-mon[23699]: pgmap v51: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:29:56.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:29:56 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:56.692 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:29:56 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:56.693 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:29:56 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:56.693 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:29:56 smithi079 ceph-mon[30023]: pgmap v50: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:29:56.693 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:29:56 smithi079 ceph-mon[30023]: pgmap v51: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:29:56.693 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:29:56 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:56.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:29:56 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:56.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:29:56 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:56.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:29:56 smithi104 ceph-mon[25639]: pgmap v50: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:29:56.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:29:56 smithi104 ceph-mon[25639]: pgmap v51: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:29:56.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:29:56 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:29:57.923 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-21T12:29:57.923 INFO:teuthology.orchestra.run.smithi079.stderr:dumped all 2024-09-21T12:29:58.037 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:29:57 smithi079 ceph-mon[23699]: pgmap v52: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:29:58.037 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:29:57 smithi079 ceph-mon[30023]: pgmap v52: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:29:58.176 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:29:57 smithi104 ceph-mon[25639]: pgmap v52: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:29:58.628 INFO:teuthology.orchestra.run.smithi079.stdout:{"pg_ready":true,"pg_map":{"version":52,"stamp":"2024-09-21T12:29:57.653366+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":3,"kb":749961216,"kb_used":226544,"kb_used_data":6292,"kb_used_omap":12,"kb_used_meta":220147,"kb_avail":749734672,"statfs":{"total":767960285184,"available":767728304128,"internally_reserved":0,"allocated":6443008,"data_stored":3677856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":225431097},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.066708"},"pg_stats":[{"pgid":"1.0","version":"59'128","reported_seq":255,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-09-21T12:28:40.392802+0000","last_change":"2024-09-21T12:28:13.865028+0000","last_active":"2024-09-21T12:28:40.392802+0000","last_peered":"2024-09-21T12:28:40.392802+0000","last_clean":"2024-09-21T12:28:40.392802+0000","last_became_active":"2024-09-21T12:28:13.864686+0000","last_became_peered":"2024-09-21T12:28:13.864686+0000","last_unstale":"2024-09-21T12:28:40.392802+0000","last_undegraded":"2024-09-21T12:28:40.392802+0000","last_fullsized":"2024-09-21T12:28:40.392802+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T12:26:19.398242+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T12:26:19.398242+0000","last_clean_scrub_stamp":"2024-09-21T12:26:19.398242+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:23:11.581508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234007,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28172,"kb_used_data":1152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716980,"statfs":{"total":95995035648,"available":95966187520,"internally_reserved":0,"allocated":1179648,"data_stored":828925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 12:29:13 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.63400000000000001}]},{"osd":1,"last update":"Sat Sep 21 12:29:13 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"last":0.58699999999999997}]},{"osd":2,"last update":"Sat Sep 21 12:29:13 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.627},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.0109999999999999,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.57199999999999995}]},{"osd":3,"last update":"Sat Sep 21 12:29:13 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.0629999999999999,"5min":1.0629999999999999,"15min":1.0629999999999999},"last":0.59699999999999998}]},{"osd":4,"last update":"Sat Sep 21 12:29:13 2024","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"last":0.443},{"interface":"front","average":{"1min":0.443,"5min":0.443,"15min":0.443},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.49299999999999999}]},{"osd":5,"last update":"Sat Sep 21 12:29:13 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":1.0429999999999999,"5min":1.0429999999999999,"15min":1.0429999999999999},"last":0.47899999999999998}]},{"osd":6,"last update":"Sat Sep 21 12:29:13 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":1.083,"5min":1.083,"15min":1.083},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.45800000000000002}]}]},{"osd":6,"up_from":46,"seq":197568495645,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28156,"kb_used_data":1140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716996,"statfs":{"total":95995035648,"available":95966203904,"internally_reserved":0,"allocated":1167360,"data_stored":828098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 12:29:52 2024","interfaces":[{"interface":"back","average":{"1min":1.623,"5min":0.71799999999999997,"15min":0.56699999999999995},"min":{"1min":0.35099999999999998,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":19.552,"5min":19.552,"15min":19.552},"last":0.50600000000000001},{"interface":"front","average":{"1min":1.617,"5min":0.749,"15min":0.60399999999999998},"min":{"1min":0.38700000000000001,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":19.523,"5min":19.523,"15min":19.523},"last":0.56799999999999995}]},{"osd":1,"last update":"Sat Sep 21 12:29:52 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.51400000000000001,"15min":0.51600000000000001},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.60999999999999999,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.52500000000000002,"15min":0.52700000000000002},"min":{"1min":0.40899999999999997,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.66200000000000003,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.42499999999999999}]},{"osd":2,"last update":"Sat Sep 21 12:29:52 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.52500000000000002,"15min":0.53100000000000003},"min":{"1min":0.39500000000000002,"5min":0.378,"15min":0.378},"max":{"1min":0.622,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.503,"15min":0.5},"min":{"1min":0.39700000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.58799999999999997}]},{"osd":3,"last update":"Sat Sep 21 12:29:52 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.54100000000000004,"15min":0.54500000000000004},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.64000000000000001,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.53500000000000003}]},{"osd":4,"last update":"Sat Sep 21 12:29:52 2024","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.68300000000000005,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.42499999999999999,"15min":0.41899999999999998},"min":{"1min":0.29999999999999999,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.72599999999999998,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.39900000000000002}]},{"osd":5,"last update":"Sat Sep 21 12:29:52 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.44600000000000001,"15min":0.442},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.71299999999999997,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.443,"15min":0.437},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.67500000000000004,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.44800000000000001}]},{"osd":7,"last update":"Sat Sep 21 12:29:11 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.496,"15min":0.496},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.63},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.64200000000000002}]}]},{"osd":5,"up_from":38,"seq":163208757281,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27592,"kb_used_data":572,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717560,"statfs":{"total":95995035648,"available":95966781440,"internally_reserved":0,"allocated":585728,"data_stored":238557,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 12:29:25 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.50700000000000001,"15min":0.51300000000000001},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.57799999999999996,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.51300000000000001,"15min":0.52100000000000002},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.61699999999999999,"5min":0.69199999999999995,"15min":0.69199999999999995},"last":0.55700000000000005}]},{"osd":1,"last update":"Sat Sep 21 12:29:25 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.53100000000000003,"15min":0.53900000000000003},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.66300000000000003,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.505,"15min":0.51100000000000001},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.54700000000000004,"5min":0.753,"15min":0.753},"last":0.49199999999999999}]},{"osd":2,"last update":"Sat Sep 21 12:29:25 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.51400000000000001,"15min":0.52000000000000002},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.66500000000000004,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.502},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.53600000000000003,"15min":0.54500000000000004},"min":{"1min":0.35699999999999998,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.63100000000000001,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.52400000000000002}]},{"osd":3,"last update":"Sat Sep 21 12:29:25 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.52200000000000002,"15min":0.52800000000000002},"min":{"1min":0.376,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.64400000000000002,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.504,"5min":0.53400000000000003,"15min":0.53900000000000003},"min":{"1min":0.38400000000000001,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.63600000000000001,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.54900000000000004}]},{"osd":4,"last update":"Sat Sep 21 12:29:25 2024","interfaces":[{"interface":"back","average":{"1min":0.39000000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"min":{"1min":0.28899999999999998,"5min":0.215,"15min":0.215},"max":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.39800000000000002,"15min":0.39600000000000002},"min":{"1min":0.28399999999999997,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.61099999999999999,"5min":0.753,"15min":0.753},"last":0.30399999999999999}]},{"osd":6,"last update":"Sat Sep 21 12:29:49 2024","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.39700000000000002,"15min":0.39100000000000001},"min":{"1min":0.30599999999999999,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.31900000000000001},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.41299999999999998,"15min":0.40500000000000003},"min":{"1min":0.28699999999999998,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.35799999999999998}]},{"osd":7,"last update":"Sat Sep 21 12:29:11 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.377}]}]},{"osd":4,"up_from":30,"seq":128849018917,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27584,"kb_used_data":564,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717568,"statfs":{"total":95995035648,"available":95966789632,"internally_reserved":0,"allocated":577536,"data_stored":237953,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 12:29:04 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.505,"15min":0.51100000000000001},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.61299999999999999,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.498,"15min":0.50600000000000001},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.60099999999999998,"5min":0.66600000000000004,"15min":0.66600000000000004},"last":0.58099999999999996}]},{"osd":1,"last update":"Sat Sep 21 12:29:04 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.47799999999999998,"15min":0.48199999999999998},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.64900000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.51400000000000001,"15min":0.51600000000000001},"min":{"1min":0.41399999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"last":0.45600000000000002}]},{"osd":2,"last update":"Sat Sep 21 12:29:04 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.51400000000000001,"15min":0.51700000000000002},"min":{"1min":0.38300000000000001,"5min":0.376,"15min":0.376},"max":{"1min":0.66300000000000003,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.46800000000000003,"5min":0.51300000000000001,"15min":0.52100000000000002},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.66800000000000004,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.48999999999999999}]},{"osd":3,"last update":"Sat Sep 21 12:29:04 2024","interfaces":[{"interface":"back","average":{"1min":0.442,"5min":0.50700000000000001,"15min":0.51800000000000002},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.69699999999999995,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.51200000000000001,"15min":0.51300000000000001},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.69499999999999995,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.56599999999999995}]},{"osd":5,"last update":"Sat Sep 21 12:29:26 2024","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.35399999999999998,"15min":0.33800000000000002},"min":{"1min":0.28999999999999998,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.39500000000000002,"15min":0.38300000000000001},"min":{"1min":0.33300000000000002,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.60399999999999998,"5min":0.66400000000000003,"15min":0.66400000000000003},"last":0.53700000000000003}]},{"osd":6,"last update":"Sat Sep 21 12:29:52 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.39300000000000002,"15min":0.38500000000000001},"min":{"1min":0.34799999999999998,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.54500000000000004,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.496,"5min":0.44,"15min":0.43099999999999999},"min":{"1min":0.36499999999999999,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.629,"5min":0.67300000000000004,"15min":0.67300000000000004},"last":0.59399999999999997}]},{"osd":7,"last update":"Sat Sep 21 12:29:13 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.60399999999999998}]}]},{"osd":3,"up_from":25,"seq":107374182442,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27592,"kb_used_data":572,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717560,"statfs":{"total":95995035648,"available":95966781440,"internally_reserved":0,"allocated":585728,"data_stored":238557,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 12:29:47 2024","interfaces":[{"interface":"back","average":{"1min":0.375,"5min":0.38100000000000001,"15min":0.379},"min":{"1min":0.247,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.51700000000000002,"5min":0.68400000000000005,"15min":0.68400000000000005},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.38300000000000001,"5min":0.40100000000000002,"15min":0.40400000000000003},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.53000000000000003,"5min":0.57699999999999996,"15min":0.57699999999999996},"last":0.34999999999999998}]},{"osd":1,"last update":"Sat Sep 21 12:29:47 2024","interfaces":[{"interface":"back","average":{"1min":0.38200000000000001,"5min":0.38800000000000001,"15min":0.38600000000000001},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"last":0.314},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.42599999999999999,"15min":0.42599999999999999},"min":{"1min":0.27400000000000002,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.59299999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"last":0.29799999999999999}]},{"osd":2,"last update":"Sat Sep 21 12:29:47 2024","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.442,"15min":0.442},"min":{"1min":0.29299999999999998,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.56200000000000006,"5min":0.624,"15min":0.624},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.42099999999999999,"15min":0.42599999999999999},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.55600000000000005,"5min":0.63900000000000001,"15min":0.63900000000000001},"last":0.26200000000000001}]},{"osd":4,"last update":"Sat Sep 21 12:29:04 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.57799999999999996,"15min":0.57599999999999996},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.497},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.59299999999999997,"15min":0.59199999999999997},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.94899999999999995,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.42099999999999999}]},{"osd":5,"last update":"Sat Sep 21 12:29:30 2024","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.621,"15min":0.63},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.56999999999999995,"15min":0.56899999999999995},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.438}]},{"osd":6,"last update":"Sat Sep 21 12:29:47 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.60899999999999999,"15min":0.60899999999999999},"min":{"1min":0.46500000000000002,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.998,"5min":0.998,"15min":0.998},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.57499999999999996,"15min":0.57299999999999995},"min":{"1min":0.36799999999999999,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.78400000000000003,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.44900000000000001}]},{"osd":7,"last update":"Sat Sep 21 12:29:12 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.48099999999999998}]}]},{"osd":2,"up_from":18,"seq":77309411375,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27592,"kb_used_data":572,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717560,"statfs":{"total":95995035648,"available":95966781440,"internally_reserved":0,"allocated":585728,"data_stored":238557,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 12:29:23 2024","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.35399999999999998,"15min":0.34599999999999997},"min":{"1min":0.23100000000000001,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.35299999999999998,"15min":0.34499999999999997},"min":{"1min":0.23899999999999999,"5min":0.214,"15min":0.214},"max":{"1min":0.54400000000000004,"5min":0.58099999999999996,"15min":0.58099999999999996},"last":0.34799999999999998}]},{"osd":1,"last update":"Sat Sep 21 12:29:23 2024","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.34899999999999998,"15min":0.34000000000000002},"min":{"1min":0.25800000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.503,"5min":0.53800000000000003,"15min":0.53800000000000003},"last":0.373},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.38200000000000001,"15min":0.376},"min":{"1min":0.19700000000000001,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.60999999999999999,"5min":0.64200000000000002,"15min":0.64200000000000002},"last":0.38600000000000001}]},{"osd":3,"last update":"Sat Sep 21 12:29:43 2024","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.42799999999999999,"15min":0.42999999999999999},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.57299999999999995,"5min":0.59099999999999997,"15min":0.59099999999999997},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.41199999999999998,"15min":0.41999999999999998},"min":{"1min":0.27000000000000002,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.51900000000000002,"5min":0.62,"15min":0.62},"last":0.30399999999999999}]},{"osd":4,"last update":"Sat Sep 21 12:29:00 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.54000000000000004,"15min":0.53800000000000003},"min":{"1min":0.35899999999999999,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.56699999999999995,"15min":0.56799999999999995},"min":{"1min":0.38300000000000001,"5min":0.379,"15min":0.379},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.53900000000000003}]},{"osd":5,"last update":"Sat Sep 21 12:29:25 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.54700000000000004,"15min":0.55400000000000005},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.53200000000000003,"15min":0.53000000000000003},"min":{"1min":0.38900000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.51100000000000001}]},{"osd":6,"last update":"Sat Sep 21 12:29:48 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.55100000000000005,"15min":0.55500000000000005},"min":{"1min":0.39800000000000002,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.70999999999999996,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.442},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.57199999999999995,"15min":0.57399999999999995},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.81899999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.48699999999999999}]},{"osd":7,"last update":"Sat Sep 21 12:29:10 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.57799999999999996}]}]},{"osd":1,"up_from":13,"seq":55834574899,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27592,"kb_used_data":572,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717560,"statfs":{"total":95995035648,"available":95966781440,"internally_reserved":0,"allocated":585728,"data_stored":238557,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 12:29:01 2024","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.33000000000000002,"15min":0.314},"min":{"1min":0.19600000000000001,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.53100000000000003,"5min":0.63300000000000001,"15min":0.63300000000000001},"last":0.28100000000000003},{"interface":"front","average":{"1min":0.32900000000000001,"5min":0.32400000000000001,"15min":0.309},"min":{"1min":0.21099999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.46700000000000003,"5min":0.56899999999999995,"15min":0.56899999999999995},"last":0.17299999999999999}]},{"osd":2,"last update":"Sat Sep 21 12:29:21 2024","interfaces":[{"interface":"back","average":{"1min":0.37,"5min":0.36199999999999999,"15min":0.35099999999999998},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.54800000000000004,"5min":0.60299999999999998,"15min":0.60299999999999998},"last":0.251},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.36399999999999999,"15min":0.35799999999999998},"min":{"1min":0.252,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.55900000000000005,"5min":0.58599999999999997,"15min":0.58599999999999997},"last":0.27200000000000002}]},{"osd":3,"last update":"Sat Sep 21 12:29:45 2024","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":0.40699999999999997,"15min":0.40699999999999997},"min":{"1min":0.29699999999999999,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.58199999999999996,"5min":0.66800000000000004,"15min":0.66800000000000004},"last":0.29299999999999998},{"interface":"front","average":{"1min":0.42499999999999999,"5min":0.41599999999999998,"15min":0.40799999999999997},"min":{"1min":0.30099999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.31}]},{"osd":4,"last update":"Sat Sep 21 12:29:01 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.55000000000000004,"15min":0.55200000000000005},"min":{"1min":0.374,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.73699999999999999,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.54200000000000004,"15min":0.54300000000000004},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.36599999999999999}]},{"osd":5,"last update":"Sat Sep 21 12:29:26 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.54800000000000004,"15min":0.54500000000000004},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.55800000000000005,"15min":0.55900000000000005},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.747,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.39000000000000001}]},{"osd":6,"last update":"Sat Sep 21 12:29:48 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.57699999999999996,"15min":0.57599999999999996},"min":{"1min":0.40200000000000002,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.57199999999999995,"15min":0.56799999999999995},"min":{"1min":0.39400000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.35799999999999998}]},{"osd":7,"last update":"Sat Sep 21 12:29:11 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.45300000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738423,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32264,"kb_used_data":1148,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":93712888,"statfs":{"total":95995035648,"available":95961997312,"internally_reserved":0,"allocated":1175552,"data_stored":828652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 21 12:29:01 2024","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.33100000000000002,"15min":0.313},"min":{"1min":0.216,"5min":0.14799999999999999,"15min":0.14799999999999999},"max":{"1min":0.497,"5min":0.55800000000000005,"15min":0.55800000000000005},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.34000000000000002,"5min":0.33900000000000002,"15min":0.32900000000000001},"min":{"1min":0.24399999999999999,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.53700000000000003,"5min":0.58999999999999997,"15min":0.58999999999999997},"last":0.317}]},{"osd":2,"last update":"Sat Sep 21 12:29:23 2024","interfaces":[{"interface":"back","average":{"1min":0.374,"5min":0.36299999999999999,"15min":0.35499999999999998},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.34200000000000003,"5min":0.35699999999999998,"15min":0.35199999999999998},"min":{"1min":0.224,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.67700000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.33800000000000002}]},{"osd":3,"last update":"Sat Sep 21 12:29:46 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.40999999999999998,"15min":0.40400000000000003},"min":{"1min":0.29699999999999999,"5min":0.252,"15min":0.252},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.42099999999999999,"5min":0.40699999999999997,"15min":0.40000000000000002},"min":{"1min":0.249,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.36899999999999999}]},{"osd":4,"last update":"Sat Sep 21 12:29:01 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.55500000000000005,"15min":0.55300000000000005},"min":{"1min":0.40699999999999997,"5min":0.316,"15min":0.316},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.437},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56899999999999995,"15min":0.56999999999999995},"min":{"1min":0.42799999999999999,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.442}]},{"osd":5,"last update":"Sat Sep 21 12:29:24 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.58599999999999997,"15min":0.59199999999999997},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.54400000000000004,"15min":0.54600000000000004},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.022,"5min":1.022,"15min":1.022},"last":0.45100000000000001}]},{"osd":6,"last update":"Sat Sep 21 12:29:46 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.57199999999999995,"15min":0.57599999999999996},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.995,"5min":0.995,"15min":0.995},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.62,"5min":0.58899999999999997,"15min":0.58399999999999996},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.127,"5min":1.127,"15min":1.127},"last":0.46800000000000003}]},{"osd":7,"last update":"Sat Sep 21 12:29:14 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.95699999999999996,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.42299999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T12:29:58.630 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-09-21T12:29:58.630 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-21T12:29:58.630 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-09-21T12:29:58.630 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell --fsid 33168022-7814-11ef-baf6-efdc52797490 -- ceph health --format=json 2024-09-21T12:29:58.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:29:58 smithi079 ceph-mon[23699]: from='client.24509 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T12:29:58.942 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:29:58 smithi079 ceph-mon[30023]: from='client.24509 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T12:29:59.090 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:29:59.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:29:58 smithi104 ceph-mon[25639]: from='client.24509 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T12:29:59.442 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Sep 21 12:29:59 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-alertmanager-a[54523]: ts=2024-09-21T12:29:59.252Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.002832147s 2024-09-21T12:29:59.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:29:59 smithi079 ceph-mon[23699]: pgmap v53: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:29:59.942 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:29:59 smithi079 ceph-mon[30023]: pgmap v53: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:30:00.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:29:59 smithi104 ceph-mon[25639]: pgmap v53: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:30:00.882 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:00 smithi079 ceph-mon[23699]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-09-21T12:30:00.882 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:00 smithi079 ceph-mon[23699]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-09-21T12:30:00.882 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:00 smithi079 ceph-mon[23699]: daemon alertmanager.a on smithi079 is in unknown state 2024-09-21T12:30:00.882 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:00 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:00.882 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:00 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:00.882 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:00 smithi079 ceph-mon[23699]: pgmap v54: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:30:00.882 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:00 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:30:00.883 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:00 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:00.883 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:00 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:30:00.883 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:00 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:30:00.883 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:00 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:00.883 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:00 smithi079 ceph-mon[30023]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-09-21T12:30:00.883 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:00 smithi079 ceph-mon[30023]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-09-21T12:30:00.883 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:00 smithi079 ceph-mon[30023]: daemon alertmanager.a on smithi079 is in unknown state 2024-09-21T12:30:00.883 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:00 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:00.883 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:00 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:00.883 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:00 smithi079 ceph-mon[30023]: pgmap v54: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:30:00.883 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:00 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:30:00.884 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:00 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:00.884 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:00 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:30:00.884 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:00 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:30:00.884 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:00 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:01.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:00 smithi104 ceph-mon[25639]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-09-21T12:30:01.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:00 smithi104 ceph-mon[25639]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-09-21T12:30:01.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:00 smithi104 ceph-mon[25639]: daemon alertmanager.a on smithi079 is in unknown state 2024-09-21T12:30:01.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:00 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:01.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:00 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:01.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:00 smithi104 ceph-mon[25639]: pgmap v54: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:30:01.178 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:00 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:30:01.178 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:00 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:01.178 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:00 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:30:01.178 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:00 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:30:01.178 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:00 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:01.554 INFO:teuthology.orchestra.run.smithi079.stdout: 2024-09-21T12:30:01.554 INFO:teuthology.orchestra.run.smithi079.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-21T12:30:01.830 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:01 smithi104 ceph-mon[25639]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-09-21T12:30:01.830 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:01 smithi104 ceph-mon[25639]: Cluster is now healthy 2024-09-21T12:30:01.830 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:01 smithi104 ceph-mon[25639]: from='client.? 172.21.15.79:0/2141159562' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-21T12:30:01.846 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:01 smithi079 ceph-mon[23699]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-09-21T12:30:01.847 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:01 smithi079 ceph-mon[23699]: Cluster is now healthy 2024-09-21T12:30:01.847 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:01 smithi079 ceph-mon[23699]: from='client.? 172.21.15.79:0/2141159562' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-21T12:30:01.847 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:01 smithi079 ceph-mon[30023]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-09-21T12:30:01.847 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:01 smithi079 ceph-mon[30023]: Cluster is now healthy 2024-09-21T12:30:01.847 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:01 smithi079 ceph-mon[30023]: from='client.? 172.21.15.79:0/2141159562' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-21T12:30:02.187 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-09-21T12:30:02.187 INFO:tasks.cephadm:Setup complete, yielding 2024-09-21T12:30:02.187 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-09-21T12:30:02.196 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi079.front.sepia.ceph.com 2024-09-21T12:30:02.196 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 33168022-7814-11ef-baf6-efdc52797490 -e sha1=441784946648ff53dd310293f9dd752afeeea72a -- bash -c 'radosgw-admin realm create --rgw-realm=r --default' 2024-09-21T12:30:02.392 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:30:02.834 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:02 smithi104 ceph-mon[25639]: pgmap v55: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:30:03.021 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:02 smithi079 ceph-mon[23699]: pgmap v55: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:30:03.022 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:02 smithi079 ceph-mon[30023]: pgmap v55: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:30:03.390 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:30:03 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: ::ffff:172.21.15.104 - - [21/Sep/2024:12:30:03] "GET /metrics HTTP/1.1" 200 26273 "" "Prometheus/2.43.0" 2024-09-21T12:30:05.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:05 smithi079 ceph-mon[23699]: pgmap v56: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:30:05.442 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:05 smithi079 ceph-mon[30023]: pgmap v56: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:30:05.602 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:05 smithi104 ceph-mon[25639]: pgmap v56: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:30:06.440 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-21T12:30:06.441 INFO:teuthology.orchestra.run.smithi079.stdout: "id": "20a300e6-70d9-4ed9-90f3-7a26a7f93062", 2024-09-21T12:30:06.441 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "r", 2024-09-21T12:30:06.441 INFO:teuthology.orchestra.run.smithi079.stdout: "current_period": "6698c38d-cba2-41f4-9e60-9162c0b7a08e", 2024-09-21T12:30:06.441 INFO:teuthology.orchestra.run.smithi079.stdout: "epoch": 1 2024-09-21T12:30:06.441 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-21T12:30:06.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:06 smithi104 ceph-mon[25639]: osdmap e60: 8 total, 8 up, 8 in 2024-09-21T12:30:06.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:06 smithi104 ceph-mon[25639]: from='client.? 172.21.15.79:0/1773102823' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-09-21T12:30:06.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:06 smithi079 ceph-mon[23699]: osdmap e60: 8 total, 8 up, 8 in 2024-09-21T12:30:06.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:06 smithi079 ceph-mon[23699]: from='client.? 172.21.15.79:0/1773102823' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-09-21T12:30:06.692 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:06 smithi079 ceph-mon[30023]: osdmap e60: 8 total, 8 up, 8 in 2024-09-21T12:30:06.692 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:06 smithi079 ceph-mon[30023]: from='client.? 172.21.15.79:0/1773102823' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-09-21T12:30:07.002 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 33168022-7814-11ef-baf6-efdc52797490 -e sha1=441784946648ff53dd310293f9dd752afeeea72a -- bash -c 'radosgw-admin zonegroup create --rgw-zonegroup=default --master --default' 2024-09-21T12:30:07.198 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:30:07.593 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:07 smithi079 ceph-mon[23699]: pgmap v58: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:30:07.593 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:07 smithi079 ceph-mon[23699]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T12:30:07.594 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:07 smithi079 ceph-mon[23699]: from='client.? 172.21.15.79:0/1773102823' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-09-21T12:30:07.594 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:07 smithi079 ceph-mon[23699]: osdmap e61: 8 total, 8 up, 8 in 2024-09-21T12:30:07.594 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:07 smithi079 ceph-mon[30023]: pgmap v58: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:30:07.594 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:07 smithi079 ceph-mon[30023]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T12:30:07.594 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:07 smithi079 ceph-mon[30023]: from='client.? 172.21.15.79:0/1773102823' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-09-21T12:30:07.594 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:07 smithi079 ceph-mon[30023]: osdmap e61: 8 total, 8 up, 8 in 2024-09-21T12:30:07.676 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:07 smithi104 ceph-mon[25639]: pgmap v58: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:30:07.676 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:07 smithi104 ceph-mon[25639]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T12:30:07.676 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:07 smithi104 ceph-mon[25639]: from='client.? 172.21.15.79:0/1773102823' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-09-21T12:30:07.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:07 smithi104 ceph-mon[25639]: osdmap e61: 8 total, 8 up, 8 in 2024-09-21T12:30:08.533 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:08 smithi079 ceph-mon[23699]: osdmap e62: 8 total, 8 up, 8 in 2024-09-21T12:30:08.533 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:08 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:08.533 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:08 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:08.533 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:08 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:08.533 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:08 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:08.533 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:08 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T12:30:08.533 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:08 smithi079 ceph-mon[30023]: osdmap e62: 8 total, 8 up, 8 in 2024-09-21T12:30:08.534 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:08 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:08.534 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:08 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:08.534 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:08 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:08.534 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:08 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:08.534 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:08 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T12:30:08.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:08 smithi104 ceph-mon[25639]: osdmap e62: 8 total, 8 up, 8 in 2024-09-21T12:30:08.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:08 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:08.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:08 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:08.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:08 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:08.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:08 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:08.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:08 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T12:30:08.725 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-21T12:30:08.725 INFO:teuthology.orchestra.run.smithi079.stdout: "id": "425d2c30-e517-4e0b-99b9-6a4308b160be", 2024-09-21T12:30:08.725 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "default", 2024-09-21T12:30:08.725 INFO:teuthology.orchestra.run.smithi079.stdout: "api_name": "default", 2024-09-21T12:30:08.725 INFO:teuthology.orchestra.run.smithi079.stdout: "is_master": true, 2024-09-21T12:30:08.726 INFO:teuthology.orchestra.run.smithi079.stdout: "endpoints": [], 2024-09-21T12:30:08.726 INFO:teuthology.orchestra.run.smithi079.stdout: "hostnames": [], 2024-09-21T12:30:08.726 INFO:teuthology.orchestra.run.smithi079.stdout: "hostnames_s3website": [], 2024-09-21T12:30:08.726 INFO:teuthology.orchestra.run.smithi079.stdout: "master_zone": "", 2024-09-21T12:30:08.726 INFO:teuthology.orchestra.run.smithi079.stdout: "zones": [], 2024-09-21T12:30:08.726 INFO:teuthology.orchestra.run.smithi079.stdout: "placement_targets": [ 2024-09-21T12:30:08.726 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-21T12:30:08.726 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "default-placement", 2024-09-21T12:30:08.726 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-09-21T12:30:08.726 INFO:teuthology.orchestra.run.smithi079.stdout: "storage_classes": [] 2024-09-21T12:30:08.726 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-21T12:30:08.726 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-09-21T12:30:08.726 INFO:teuthology.orchestra.run.smithi079.stdout: "default_placement": "default-placement", 2024-09-21T12:30:08.726 INFO:teuthology.orchestra.run.smithi079.stdout: "realm_id": "20a300e6-70d9-4ed9-90f3-7a26a7f93062", 2024-09-21T12:30:08.726 INFO:teuthology.orchestra.run.smithi079.stdout: "sync_policy": { 2024-09-21T12:30:08.727 INFO:teuthology.orchestra.run.smithi079.stdout: "groups": [] 2024-09-21T12:30:08.727 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-21T12:30:08.727 INFO:teuthology.orchestra.run.smithi079.stdout: "enabled_features": [ 2024-09-21T12:30:08.727 INFO:teuthology.orchestra.run.smithi079.stdout: "resharding" 2024-09-21T12:30:08.727 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-21T12:30:08.727 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-21T12:30:09.274 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 33168022-7814-11ef-baf6-efdc52797490 -e sha1=441784946648ff53dd310293f9dd752afeeea72a -- bash -c 'radosgw-admin zone create --rgw-zonegroup=default --rgw-zone=z --master --default' 2024-09-21T12:30:09.470 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:30:09.590 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:09 smithi079 ceph-mon[23699]: pgmap v61: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:30:09.590 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:09 smithi079 ceph-mon[23699]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T12:30:09.590 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:09 smithi079 ceph-mon[23699]: Cluster is now healthy 2024-09-21T12:30:09.590 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:09 smithi079 ceph-mon[30023]: pgmap v61: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:30:09.590 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:09 smithi079 ceph-mon[30023]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T12:30:09.590 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:09 smithi079 ceph-mon[30023]: Cluster is now healthy 2024-09-21T12:30:09.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:09 smithi104 ceph-mon[25639]: pgmap v61: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:30:09.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:09 smithi104 ceph-mon[25639]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T12:30:09.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:09 smithi104 ceph-mon[25639]: Cluster is now healthy 2024-09-21T12:30:11.041 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-21T12:30:11.041 INFO:teuthology.orchestra.run.smithi079.stdout: "id": "b866b23c-447d-449b-8de0-6b1482fa5a18", 2024-09-21T12:30:11.041 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "z", 2024-09-21T12:30:11.041 INFO:teuthology.orchestra.run.smithi079.stdout: "domain_root": "z.rgw.meta:root", 2024-09-21T12:30:11.041 INFO:teuthology.orchestra.run.smithi079.stdout: "control_pool": "z.rgw.control", 2024-09-21T12:30:11.041 INFO:teuthology.orchestra.run.smithi079.stdout: "gc_pool": "z.rgw.log:gc", 2024-09-21T12:30:11.041 INFO:teuthology.orchestra.run.smithi079.stdout: "lc_pool": "z.rgw.log:lc", 2024-09-21T12:30:11.042 INFO:teuthology.orchestra.run.smithi079.stdout: "log_pool": "z.rgw.log", 2024-09-21T12:30:11.042 INFO:teuthology.orchestra.run.smithi079.stdout: "intent_log_pool": "z.rgw.log:intent", 2024-09-21T12:30:11.042 INFO:teuthology.orchestra.run.smithi079.stdout: "usage_log_pool": "z.rgw.log:usage", 2024-09-21T12:30:11.042 INFO:teuthology.orchestra.run.smithi079.stdout: "roles_pool": "z.rgw.meta:roles", 2024-09-21T12:30:11.042 INFO:teuthology.orchestra.run.smithi079.stdout: "reshard_pool": "z.rgw.log:reshard", 2024-09-21T12:30:11.042 INFO:teuthology.orchestra.run.smithi079.stdout: "user_keys_pool": "z.rgw.meta:users.keys", 2024-09-21T12:30:11.042 INFO:teuthology.orchestra.run.smithi079.stdout: "user_email_pool": "z.rgw.meta:users.email", 2024-09-21T12:30:11.042 INFO:teuthology.orchestra.run.smithi079.stdout: "user_swift_pool": "z.rgw.meta:users.swift", 2024-09-21T12:30:11.042 INFO:teuthology.orchestra.run.smithi079.stdout: "user_uid_pool": "z.rgw.meta:users.uid", 2024-09-21T12:30:11.042 INFO:teuthology.orchestra.run.smithi079.stdout: "otp_pool": "z.rgw.otp", 2024-09-21T12:30:11.042 INFO:teuthology.orchestra.run.smithi079.stdout: "system_key": { 2024-09-21T12:30:11.042 INFO:teuthology.orchestra.run.smithi079.stdout: "access_key": "", 2024-09-21T12:30:11.042 INFO:teuthology.orchestra.run.smithi079.stdout: "secret_key": "" 2024-09-21T12:30:11.042 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-21T12:30:11.042 INFO:teuthology.orchestra.run.smithi079.stdout: "placement_pools": [ 2024-09-21T12:30:11.043 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-21T12:30:11.043 INFO:teuthology.orchestra.run.smithi079.stdout: "key": "default-placement", 2024-09-21T12:30:11.043 INFO:teuthology.orchestra.run.smithi079.stdout: "val": { 2024-09-21T12:30:11.043 INFO:teuthology.orchestra.run.smithi079.stdout: "index_pool": "z.rgw.buckets.index", 2024-09-21T12:30:11.043 INFO:teuthology.orchestra.run.smithi079.stdout: "storage_classes": { 2024-09-21T12:30:11.043 INFO:teuthology.orchestra.run.smithi079.stdout: "STANDARD": { 2024-09-21T12:30:11.043 INFO:teuthology.orchestra.run.smithi079.stdout: "data_pool": "z.rgw.buckets.data" 2024-09-21T12:30:11.043 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-21T12:30:11.043 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-21T12:30:11.043 INFO:teuthology.orchestra.run.smithi079.stdout: "data_extra_pool": "z.rgw.buckets.non-ec", 2024-09-21T12:30:11.043 INFO:teuthology.orchestra.run.smithi079.stdout: "index_type": 0, 2024-09-21T12:30:11.043 INFO:teuthology.orchestra.run.smithi079.stdout: "inline_data": true 2024-09-21T12:30:11.043 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-21T12:30:11.044 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-21T12:30:11.044 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-09-21T12:30:11.044 INFO:teuthology.orchestra.run.smithi079.stdout: "realm_id": "20a300e6-70d9-4ed9-90f3-7a26a7f93062", 2024-09-21T12:30:11.044 INFO:teuthology.orchestra.run.smithi079.stdout: "notif_pool": "z.rgw.log:notif" 2024-09-21T12:30:11.044 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-21T12:30:11.680 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:11 smithi079 ceph-mon[23699]: pgmap v62: 33 pgs: 33 active+clean; 578 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 2.0 KiB/s wr, 4 op/s 2024-09-21T12:30:11.680 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:11 smithi079 ceph-mon[30023]: pgmap v62: 33 pgs: 33 active+clean; 578 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 2.0 KiB/s wr, 4 op/s 2024-09-21T12:30:11.702 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 33168022-7814-11ef-baf6-efdc52797490 -e sha1=441784946648ff53dd310293f9dd752afeeea72a -- bash -c 'radosgw-admin period update --rgw-realm=r --commit' 2024-09-21T12:30:11.896 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:30:11.926 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:11 smithi104 ceph-mon[25639]: pgmap v62: 33 pgs: 33 active+clean; 578 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 2.0 KiB/s wr, 4 op/s 2024-09-21T12:30:13.433 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:30:13 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: ::ffff:172.21.15.104 - - [21/Sep/2024:12:30:13] "GET /metrics HTTP/1.1" 200 28631 "" "Prometheus/2.43.0" 2024-09-21T12:30:13.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:13 smithi079 ceph-mon[23699]: pgmap v63: 33 pgs: 33 active+clean; 578 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 765 B/s rd, 1.8 KiB/s wr, 3 op/s 2024-09-21T12:30:13.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:13 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:13.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:13 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:13.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:13 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:13.692 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:13 smithi079 ceph-mon[30023]: pgmap v63: 33 pgs: 33 active+clean; 578 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 765 B/s rd, 1.8 KiB/s wr, 3 op/s 2024-09-21T12:30:13.692 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:13 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:13.692 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:13 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:13.692 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:13 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:13.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:13 smithi104 ceph-mon[25639]: pgmap v63: 33 pgs: 33 active+clean; 578 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 765 B/s rd, 1.8 KiB/s wr, 3 op/s 2024-09-21T12:30:13.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:13 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:13.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:13 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:13.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:13 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:15.518 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:15 smithi079 ceph-mon[23699]: pgmap v64: 33 pgs: 33 active+clean; 578 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 1.5 KiB/s wr, 3 op/s 2024-09-21T12:30:15.519 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:15 smithi079 ceph-mon[23699]: osdmap e63: 8 total, 8 up, 8 in 2024-09-21T12:30:15.519 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:15 smithi079 ceph-mon[23699]: from='client.? 172.21.15.79:0/855836373' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-09-21T12:30:15.519 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:15 smithi079 ceph-mon[23699]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-09-21T12:30:15.519 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:15 smithi079 ceph-mon[30023]: pgmap v64: 33 pgs: 33 active+clean; 578 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 1.5 KiB/s wr, 3 op/s 2024-09-21T12:30:15.519 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:15 smithi079 ceph-mon[30023]: osdmap e63: 8 total, 8 up, 8 in 2024-09-21T12:30:15.519 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:15 smithi079 ceph-mon[30023]: from='client.? 172.21.15.79:0/855836373' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-09-21T12:30:15.519 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:15 smithi079 ceph-mon[30023]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-09-21T12:30:15.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:15 smithi104 ceph-mon[25639]: pgmap v64: 33 pgs: 33 active+clean; 578 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 1.5 KiB/s wr, 3 op/s 2024-09-21T12:30:15.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:15 smithi104 ceph-mon[25639]: osdmap e63: 8 total, 8 up, 8 in 2024-09-21T12:30:15.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:15 smithi104 ceph-mon[25639]: from='client.? 172.21.15.79:0/855836373' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-09-21T12:30:15.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:15 smithi104 ceph-mon[25639]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-09-21T12:30:16.683 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:16 smithi079 ceph-mon[23699]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]': finished 2024-09-21T12:30:16.683 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:16 smithi079 ceph-mon[23699]: osdmap e64: 8 total, 8 up, 8 in 2024-09-21T12:30:16.683 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:16 smithi079 ceph-mon[30023]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]': finished 2024-09-21T12:30:16.683 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:16 smithi079 ceph-mon[30023]: osdmap e64: 8 total, 8 up, 8 in 2024-09-21T12:30:16.926 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:16 smithi104 ceph-mon[25639]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]': finished 2024-09-21T12:30:16.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:16 smithi104 ceph-mon[25639]: osdmap e64: 8 total, 8 up, 8 in 2024-09-21T12:30:17.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:17 smithi104 ceph-mon[25639]: pgmap v67: 65 pgs: 32 unknown, 33 active+clean; 579 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 3.6 KiB/s rd, 2.1 KiB/s wr, 7 op/s 2024-09-21T12:30:17.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:17 smithi104 ceph-mon[25639]: osdmap e65: 8 total, 8 up, 8 in 2024-09-21T12:30:17.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:17 smithi104 ceph-mon[25639]: from='client.? 172.21.15.79:0/855836373' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-09-21T12:30:17.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:17 smithi104 ceph-mon[25639]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-09-21T12:30:17.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:17 smithi079 ceph-mon[23699]: pgmap v67: 65 pgs: 32 unknown, 33 active+clean; 579 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 3.6 KiB/s rd, 2.1 KiB/s wr, 7 op/s 2024-09-21T12:30:17.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:17 smithi079 ceph-mon[23699]: osdmap e65: 8 total, 8 up, 8 in 2024-09-21T12:30:17.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:17 smithi079 ceph-mon[23699]: from='client.? 172.21.15.79:0/855836373' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-09-21T12:30:17.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:17 smithi079 ceph-mon[23699]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-09-21T12:30:17.942 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:17 smithi079 ceph-mon[30023]: pgmap v67: 65 pgs: 32 unknown, 33 active+clean; 579 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 3.6 KiB/s rd, 2.1 KiB/s wr, 7 op/s 2024-09-21T12:30:17.942 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:17 smithi079 ceph-mon[30023]: osdmap e65: 8 total, 8 up, 8 in 2024-09-21T12:30:17.942 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:17 smithi079 ceph-mon[30023]: from='client.? 172.21.15.79:0/855836373' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-09-21T12:30:17.942 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:17 smithi079 ceph-mon[30023]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-09-21T12:30:18.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:18 smithi104 ceph-mon[25639]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]': finished 2024-09-21T12:30:18.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:18 smithi104 ceph-mon[25639]: osdmap e66: 8 total, 8 up, 8 in 2024-09-21T12:30:18.941 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:18 smithi079 ceph-mon[23699]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]': finished 2024-09-21T12:30:18.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:18 smithi079 ceph-mon[23699]: osdmap e66: 8 total, 8 up, 8 in 2024-09-21T12:30:18.942 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:18 smithi079 ceph-mon[30023]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]': finished 2024-09-21T12:30:18.942 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:18 smithi079 ceph-mon[30023]: osdmap e66: 8 total, 8 up, 8 in 2024-09-21T12:30:19.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:19 smithi104 ceph-mon[25639]: pgmap v70: 97 pgs: 64 unknown, 33 active+clean; 579 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 6.0 KiB/s rd, 1.2 KiB/s wr, 8 op/s 2024-09-21T12:30:19.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:19 smithi104 ceph-mon[25639]: osdmap e67: 8 total, 8 up, 8 in 2024-09-21T12:30:19.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:19 smithi104 ceph-mon[25639]: from='client.? 172.21.15.79:0/2692100201' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-09-21T12:30:19.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:19 smithi104 ceph-mon[25639]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-09-21T12:30:19.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:19 smithi079 ceph-mon[23699]: pgmap v70: 97 pgs: 64 unknown, 33 active+clean; 579 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 6.0 KiB/s rd, 1.2 KiB/s wr, 8 op/s 2024-09-21T12:30:19.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:19 smithi079 ceph-mon[23699]: osdmap e67: 8 total, 8 up, 8 in 2024-09-21T12:30:19.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:19 smithi079 ceph-mon[23699]: from='client.? 172.21.15.79:0/2692100201' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-09-21T12:30:19.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:19 smithi079 ceph-mon[23699]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-09-21T12:30:19.942 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:19 smithi079 ceph-mon[30023]: pgmap v70: 97 pgs: 64 unknown, 33 active+clean; 579 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 6.0 KiB/s rd, 1.2 KiB/s wr, 8 op/s 2024-09-21T12:30:19.942 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:19 smithi079 ceph-mon[30023]: osdmap e67: 8 total, 8 up, 8 in 2024-09-21T12:30:19.942 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:19 smithi079 ceph-mon[30023]: from='client.? 172.21.15.79:0/2692100201' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-09-21T12:30:19.942 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:19 smithi079 ceph-mon[30023]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-09-21T12:30:20.543 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-21T12:30:20.543 INFO:teuthology.orchestra.run.smithi079.stdout: "id": "d36f3da0-e1db-4960-8f10-3c27266b8442", 2024-09-21T12:30:20.543 INFO:teuthology.orchestra.run.smithi079.stdout: "epoch": 1, 2024-09-21T12:30:20.543 INFO:teuthology.orchestra.run.smithi079.stdout: "predecessor_uuid": "6698c38d-cba2-41f4-9e60-9162c0b7a08e", 2024-09-21T12:30:20.543 INFO:teuthology.orchestra.run.smithi079.stdout: "sync_status": [], 2024-09-21T12:30:20.543 INFO:teuthology.orchestra.run.smithi079.stdout: "period_map": { 2024-09-21T12:30:20.543 INFO:teuthology.orchestra.run.smithi079.stdout: "id": "d36f3da0-e1db-4960-8f10-3c27266b8442", 2024-09-21T12:30:20.543 INFO:teuthology.orchestra.run.smithi079.stdout: "zonegroups": [ 2024-09-21T12:30:20.543 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-21T12:30:20.543 INFO:teuthology.orchestra.run.smithi079.stdout: "id": "425d2c30-e517-4e0b-99b9-6a4308b160be", 2024-09-21T12:30:20.544 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "default", 2024-09-21T12:30:20.544 INFO:teuthology.orchestra.run.smithi079.stdout: "api_name": "default", 2024-09-21T12:30:20.544 INFO:teuthology.orchestra.run.smithi079.stdout: "is_master": true, 2024-09-21T12:30:20.544 INFO:teuthology.orchestra.run.smithi079.stdout: "endpoints": [], 2024-09-21T12:30:20.544 INFO:teuthology.orchestra.run.smithi079.stdout: "hostnames": [], 2024-09-21T12:30:20.544 INFO:teuthology.orchestra.run.smithi079.stdout: "hostnames_s3website": [], 2024-09-21T12:30:20.544 INFO:teuthology.orchestra.run.smithi079.stdout: "master_zone": "b866b23c-447d-449b-8de0-6b1482fa5a18", 2024-09-21T12:30:20.544 INFO:teuthology.orchestra.run.smithi079.stdout: "zones": [ 2024-09-21T12:30:20.544 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-21T12:30:20.544 INFO:teuthology.orchestra.run.smithi079.stdout: "id": "b866b23c-447d-449b-8de0-6b1482fa5a18", 2024-09-21T12:30:20.544 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "z", 2024-09-21T12:30:20.544 INFO:teuthology.orchestra.run.smithi079.stdout: "endpoints": [], 2024-09-21T12:30:20.544 INFO:teuthology.orchestra.run.smithi079.stdout: "log_meta": false, 2024-09-21T12:30:20.544 INFO:teuthology.orchestra.run.smithi079.stdout: "log_data": false, 2024-09-21T12:30:20.544 INFO:teuthology.orchestra.run.smithi079.stdout: "bucket_index_max_shards": 11, 2024-09-21T12:30:20.544 INFO:teuthology.orchestra.run.smithi079.stdout: "read_only": false, 2024-09-21T12:30:20.544 INFO:teuthology.orchestra.run.smithi079.stdout: "tier_type": "", 2024-09-21T12:30:20.545 INFO:teuthology.orchestra.run.smithi079.stdout: "sync_from_all": true, 2024-09-21T12:30:20.545 INFO:teuthology.orchestra.run.smithi079.stdout: "sync_from": [], 2024-09-21T12:30:20.545 INFO:teuthology.orchestra.run.smithi079.stdout: "redirect_zone": "", 2024-09-21T12:30:20.545 INFO:teuthology.orchestra.run.smithi079.stdout: "supported_features": [ 2024-09-21T12:30:20.545 INFO:teuthology.orchestra.run.smithi079.stdout: "compress-encrypted", 2024-09-21T12:30:20.545 INFO:teuthology.orchestra.run.smithi079.stdout: "resharding" 2024-09-21T12:30:20.545 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-21T12:30:20.545 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-21T12:30:20.545 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-09-21T12:30:20.545 INFO:teuthology.orchestra.run.smithi079.stdout: "placement_targets": [ 2024-09-21T12:30:20.545 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-21T12:30:20.545 INFO:teuthology.orchestra.run.smithi079.stdout: "name": "default-placement", 2024-09-21T12:30:20.545 INFO:teuthology.orchestra.run.smithi079.stdout: "tags": [], 2024-09-21T12:30:20.545 INFO:teuthology.orchestra.run.smithi079.stdout: "storage_classes": [ 2024-09-21T12:30:20.545 INFO:teuthology.orchestra.run.smithi079.stdout: "STANDARD" 2024-09-21T12:30:20.545 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-21T12:30:20.545 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-21T12:30:20.546 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-09-21T12:30:20.546 INFO:teuthology.orchestra.run.smithi079.stdout: "default_placement": "default-placement", 2024-09-21T12:30:20.546 INFO:teuthology.orchestra.run.smithi079.stdout: "realm_id": "20a300e6-70d9-4ed9-90f3-7a26a7f93062", 2024-09-21T12:30:20.546 INFO:teuthology.orchestra.run.smithi079.stdout: "sync_policy": { 2024-09-21T12:30:20.546 INFO:teuthology.orchestra.run.smithi079.stdout: "groups": [] 2024-09-21T12:30:20.546 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-21T12:30:20.546 INFO:teuthology.orchestra.run.smithi079.stdout: "enabled_features": [ 2024-09-21T12:30:20.546 INFO:teuthology.orchestra.run.smithi079.stdout: "resharding" 2024-09-21T12:30:20.546 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-21T12:30:20.546 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-21T12:30:20.546 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-09-21T12:30:20.546 INFO:teuthology.orchestra.run.smithi079.stdout: "short_zone_ids": [ 2024-09-21T12:30:20.546 INFO:teuthology.orchestra.run.smithi079.stdout: { 2024-09-21T12:30:20.546 INFO:teuthology.orchestra.run.smithi079.stdout: "key": "b866b23c-447d-449b-8de0-6b1482fa5a18", 2024-09-21T12:30:20.546 INFO:teuthology.orchestra.run.smithi079.stdout: "val": 1874471932 2024-09-21T12:30:20.546 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-21T12:30:20.547 INFO:teuthology.orchestra.run.smithi079.stdout: ] 2024-09-21T12:30:20.547 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-21T12:30:20.547 INFO:teuthology.orchestra.run.smithi079.stdout: "master_zonegroup": "425d2c30-e517-4e0b-99b9-6a4308b160be", 2024-09-21T12:30:20.547 INFO:teuthology.orchestra.run.smithi079.stdout: "master_zone": "b866b23c-447d-449b-8de0-6b1482fa5a18", 2024-09-21T12:30:20.547 INFO:teuthology.orchestra.run.smithi079.stdout: "period_config": { 2024-09-21T12:30:20.547 INFO:teuthology.orchestra.run.smithi079.stdout: "bucket_quota": { 2024-09-21T12:30:20.547 INFO:teuthology.orchestra.run.smithi079.stdout: "enabled": false, 2024-09-21T12:30:20.547 INFO:teuthology.orchestra.run.smithi079.stdout: "check_on_raw": false, 2024-09-21T12:30:20.547 INFO:teuthology.orchestra.run.smithi079.stdout: "max_size": -1, 2024-09-21T12:30:20.547 INFO:teuthology.orchestra.run.smithi079.stdout: "max_size_kb": 0, 2024-09-21T12:30:20.547 INFO:teuthology.orchestra.run.smithi079.stdout: "max_objects": -1 2024-09-21T12:30:20.547 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-21T12:30:20.547 INFO:teuthology.orchestra.run.smithi079.stdout: "user_quota": { 2024-09-21T12:30:20.547 INFO:teuthology.orchestra.run.smithi079.stdout: "enabled": false, 2024-09-21T12:30:20.547 INFO:teuthology.orchestra.run.smithi079.stdout: "check_on_raw": false, 2024-09-21T12:30:20.548 INFO:teuthology.orchestra.run.smithi079.stdout: "max_size": -1, 2024-09-21T12:30:20.548 INFO:teuthology.orchestra.run.smithi079.stdout: "max_size_kb": 0, 2024-09-21T12:30:20.548 INFO:teuthology.orchestra.run.smithi079.stdout: "max_objects": -1 2024-09-21T12:30:20.548 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-21T12:30:20.548 INFO:teuthology.orchestra.run.smithi079.stdout: "user_ratelimit": { 2024-09-21T12:30:20.548 INFO:teuthology.orchestra.run.smithi079.stdout: "max_read_ops": 0, 2024-09-21T12:30:20.548 INFO:teuthology.orchestra.run.smithi079.stdout: "max_write_ops": 0, 2024-09-21T12:30:20.548 INFO:teuthology.orchestra.run.smithi079.stdout: "max_read_bytes": 0, 2024-09-21T12:30:20.548 INFO:teuthology.orchestra.run.smithi079.stdout: "max_write_bytes": 0, 2024-09-21T12:30:20.548 INFO:teuthology.orchestra.run.smithi079.stdout: "enabled": false 2024-09-21T12:30:20.548 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-21T12:30:20.548 INFO:teuthology.orchestra.run.smithi079.stdout: "bucket_ratelimit": { 2024-09-21T12:30:20.548 INFO:teuthology.orchestra.run.smithi079.stdout: "max_read_ops": 0, 2024-09-21T12:30:20.548 INFO:teuthology.orchestra.run.smithi079.stdout: "max_write_ops": 0, 2024-09-21T12:30:20.549 INFO:teuthology.orchestra.run.smithi079.stdout: "max_read_bytes": 0, 2024-09-21T12:30:20.549 INFO:teuthology.orchestra.run.smithi079.stdout: "max_write_bytes": 0, 2024-09-21T12:30:20.549 INFO:teuthology.orchestra.run.smithi079.stdout: "enabled": false 2024-09-21T12:30:20.549 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-21T12:30:20.549 INFO:teuthology.orchestra.run.smithi079.stdout: "anonymous_ratelimit": { 2024-09-21T12:30:20.549 INFO:teuthology.orchestra.run.smithi079.stdout: "max_read_ops": 0, 2024-09-21T12:30:20.549 INFO:teuthology.orchestra.run.smithi079.stdout: "max_write_ops": 0, 2024-09-21T12:30:20.549 INFO:teuthology.orchestra.run.smithi079.stdout: "max_read_bytes": 0, 2024-09-21T12:30:20.549 INFO:teuthology.orchestra.run.smithi079.stdout: "max_write_bytes": 0, 2024-09-21T12:30:20.549 INFO:teuthology.orchestra.run.smithi079.stdout: "enabled": false 2024-09-21T12:30:20.549 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-21T12:30:20.549 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-21T12:30:20.549 INFO:teuthology.orchestra.run.smithi079.stdout: "realm_id": "20a300e6-70d9-4ed9-90f3-7a26a7f93062", 2024-09-21T12:30:20.549 INFO:teuthology.orchestra.run.smithi079.stdout: "realm_name": "r", 2024-09-21T12:30:20.549 INFO:teuthology.orchestra.run.smithi079.stdout: "realm_epoch": 2 2024-09-21T12:30:20.549 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-21T12:30:20.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:20 smithi104 ceph-mon[25639]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]': finished 2024-09-21T12:30:20.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:20 smithi104 ceph-mon[25639]: osdmap e68: 8 total, 8 up, 8 in 2024-09-21T12:30:20.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:20 smithi104 ceph-mon[25639]: from='client.? 172.21.15.79:0/2692100201' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-09-21T12:30:20.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:20 smithi104 ceph-mon[25639]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-09-21T12:30:20.819 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:20 smithi079 ceph-mon[23699]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]': finished 2024-09-21T12:30:20.820 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:20 smithi079 ceph-mon[23699]: osdmap e68: 8 total, 8 up, 8 in 2024-09-21T12:30:20.820 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:20 smithi079 ceph-mon[23699]: from='client.? 172.21.15.79:0/2692100201' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-09-21T12:30:20.820 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:20 smithi079 ceph-mon[23699]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-09-21T12:30:20.820 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:20 smithi079 ceph-mon[30023]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]': finished 2024-09-21T12:30:20.820 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:20 smithi079 ceph-mon[30023]: osdmap e68: 8 total, 8 up, 8 in 2024-09-21T12:30:20.820 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:20 smithi079 ceph-mon[30023]: from='client.? 172.21.15.79:0/2692100201' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-09-21T12:30:20.820 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:20 smithi079 ceph-mon[30023]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-09-21T12:30:21.256 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 33168022-7814-11ef-baf6-efdc52797490 -e sha1=441784946648ff53dd310293f9dd752afeeea72a -- bash -c 'ceph orch apply rgw foo --realm r --zone z --placement=2 --port=8000' 2024-09-21T12:30:21.461 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:30:21.765 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:21 smithi079 ceph-mon[23699]: pgmap v73: 129 pgs: 32 creating+peering, 97 active+clean; 579 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 511 B/s wr, 1 op/s 2024-09-21T12:30:21.765 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:21 smithi079 ceph-mon[23699]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-21T12:30:21.765 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:21 smithi079 ceph-mon[23699]: osdmap e69: 8 total, 8 up, 8 in 2024-09-21T12:30:21.765 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:21 smithi079 ceph-mon[30023]: pgmap v73: 129 pgs: 32 creating+peering, 97 active+clean; 579 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 511 B/s wr, 1 op/s 2024-09-21T12:30:21.765 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:21 smithi079 ceph-mon[30023]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-21T12:30:21.765 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:21 smithi079 ceph-mon[30023]: osdmap e69: 8 total, 8 up, 8 in 2024-09-21T12:30:21.787 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:21 smithi104 ceph-mon[25639]: pgmap v73: 129 pgs: 32 creating+peering, 97 active+clean; 579 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 511 B/s wr, 1 op/s 2024-09-21T12:30:21.787 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:21 smithi104 ceph-mon[25639]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-21T12:30:21.787 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:21 smithi104 ceph-mon[25639]: osdmap e69: 8 total, 8 up, 8 in 2024-09-21T12:30:23.192 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:30:23 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: ::ffff:172.21.15.104 - - [21/Sep/2024:12:30:23] "GET /metrics HTTP/1.1" 200 28631 "" "Prometheus/2.43.0" 2024-09-21T12:30:23.296 INFO:teuthology.orchestra.run.smithi079.stdout:Scheduled rgw.foo update... 2024-09-21T12:30:23.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:23 smithi104 ceph-mon[25639]: pgmap v75: 129 pgs: 32 creating+peering, 97 active+clean; 579 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 220 B/s rd, 441 B/s wr, 1 op/s 2024-09-21T12:30:23.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:23 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:23.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:23 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:30:23.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:23 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:30:23.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:23 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:30:23.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:23 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:23.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:23 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi104.qonbxe", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-21T12:30:23.678 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:23 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi104.qonbxe", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-21T12:30:23.678 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:23 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:23.678 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:23 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:30:23.678 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:23 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:23.678 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:23 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T12:30:23.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:23 smithi079 ceph-mon[23699]: pgmap v75: 129 pgs: 32 creating+peering, 97 active+clean; 579 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 220 B/s rd, 441 B/s wr, 1 op/s 2024-09-21T12:30:23.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:23 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:23.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:23 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:30:23.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:23 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:30:23.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:23 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:30:23.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:23 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:23.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:23 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi104.qonbxe", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-21T12:30:23.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:23 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi104.qonbxe", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-21T12:30:23.943 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:23 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:23.943 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:23 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:30:23.943 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:23 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:23.943 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:23 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T12:30:23.943 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:23 smithi079 ceph-mon[30023]: pgmap v75: 129 pgs: 32 creating+peering, 97 active+clean; 579 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 220 B/s rd, 441 B/s wr, 1 op/s 2024-09-21T12:30:23.943 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:23 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:23.943 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:23 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:30:23.943 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:23 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:30:23.943 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:23 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:30:23.943 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:23 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:23.943 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:23 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi104.qonbxe", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-21T12:30:23.943 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:23 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi104.qonbxe", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-21T12:30:23.944 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:23 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:23.944 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:23 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:30:23.944 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:23 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:23.944 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:23 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T12:30:24.010 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 33168022-7814-11ef-baf6-efdc52797490 -e sha1=441784946648ff53dd310293f9dd752afeeea72a -- bash -c 'ceph orch apply rgw smpl' 2024-09-21T12:30:24.209 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:30:24.664 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:24 smithi079 ceph-mon[23699]: from='client.24605 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo", "realm": "r", "zone": "z", "placement": "2", "port": 8000, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:30:24.664 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:24 smithi079 ceph-mon[23699]: Saving service rgw.foo spec with placement count:2 2024-09-21T12:30:24.664 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:24 smithi079 ceph-mon[23699]: Deploying daemon rgw.foo.smithi104.qonbxe on smithi104 2024-09-21T12:30:24.665 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:24 smithi079 ceph-mon[30023]: from='client.24605 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo", "realm": "r", "zone": "z", "placement": "2", "port": 8000, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:30:24.665 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:24 smithi079 ceph-mon[30023]: Saving service rgw.foo spec with placement count:2 2024-09-21T12:30:24.665 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:24 smithi079 ceph-mon[30023]: Deploying daemon rgw.foo.smithi104.qonbxe on smithi104 2024-09-21T12:30:24.763 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:24 smithi104 ceph-mon[25639]: from='client.24605 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo", "realm": "r", "zone": "z", "placement": "2", "port": 8000, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:30:24.763 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:24 smithi104 ceph-mon[25639]: Saving service rgw.foo spec with placement count:2 2024-09-21T12:30:24.763 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:24 smithi104 ceph-mon[25639]: Deploying daemon rgw.foo.smithi104.qonbxe on smithi104 2024-09-21T12:30:25.657 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:25 smithi079 ceph-mon[23699]: pgmap v76: 129 pgs: 32 creating+peering, 97 active+clean; 579 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 341 B/s wr, 0 op/s 2024-09-21T12:30:25.657 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:25 smithi079 ceph-mon[30023]: pgmap v76: 129 pgs: 32 creating+peering, 97 active+clean; 579 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 341 B/s wr, 0 op/s 2024-09-21T12:30:25.676 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:25 smithi104 ceph-mon[25639]: pgmap v76: 129 pgs: 32 creating+peering, 97 active+clean; 579 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 341 B/s wr, 0 op/s 2024-09-21T12:30:26.151 INFO:teuthology.orchestra.run.smithi079.stdout:Scheduled rgw.smpl update... 2024-09-21T12:30:26.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:26 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:26.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:26 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:26.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:26 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:26.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:26 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi079.dfrddk", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-21T12:30:26.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:26 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi079.dfrddk", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-21T12:30:26.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:26 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:26.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:26 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:30:26.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:26 smithi104 ceph-mon[25639]: Deploying daemon rgw.foo.smithi079.dfrddk on smithi079 2024-09-21T12:30:26.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:26 smithi104 ceph-mon[25639]: pgmap v77: 129 pgs: 129 active+clean; 581 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 1.6 KiB/s wr, 4 op/s 2024-09-21T12:30:26.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:26 smithi104 ceph-mon[25639]: from='client.24544 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "smpl", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:30:26.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:26 smithi104 ceph-mon[25639]: Saving service rgw.smpl spec with placement count:2 2024-09-21T12:30:26.928 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:26 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:27.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:26 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:27.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:26 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:27.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:26 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:27.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:26 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi079.dfrddk", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-21T12:30:27.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:26 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi079.dfrddk", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-21T12:30:27.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:26 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:27.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:26 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:30:27.193 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:26 smithi079 ceph-mon[30023]: Deploying daemon rgw.foo.smithi079.dfrddk on smithi079 2024-09-21T12:30:27.193 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:26 smithi079 ceph-mon[30023]: pgmap v77: 129 pgs: 129 active+clean; 581 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 1.6 KiB/s wr, 4 op/s 2024-09-21T12:30:27.193 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:26 smithi079 ceph-mon[30023]: from='client.24544 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "smpl", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:30:27.193 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:26 smithi079 ceph-mon[30023]: Saving service rgw.smpl spec with placement count:2 2024-09-21T12:30:27.193 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:26 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:27.193 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:26 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:27.193 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:26 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:27.193 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:26 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:27.193 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:26 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi079.dfrddk", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-21T12:30:27.193 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:26 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi079.dfrddk", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-21T12:30:27.193 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:26 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:27.193 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:26 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:30:27.193 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:26 smithi079 ceph-mon[23699]: Deploying daemon rgw.foo.smithi079.dfrddk on smithi079 2024-09-21T12:30:27.194 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:26 smithi079 ceph-mon[23699]: pgmap v77: 129 pgs: 129 active+clean; 581 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 1.6 KiB/s wr, 4 op/s 2024-09-21T12:30:27.194 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:26 smithi079 ceph-mon[23699]: from='client.24544 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "smpl", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:30:27.194 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:26 smithi079 ceph-mon[23699]: Saving service rgw.smpl spec with placement count:2 2024-09-21T12:30:27.194 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:26 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:28.200 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 33168022-7814-11ef-baf6-efdc52797490 -e sha1=441784946648ff53dd310293f9dd752afeeea72a -- bash -c 'ceph osd pool create foo' 2024-09-21T12:30:28.763 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:30:28.894 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:28 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:28.894 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:28 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:28.894 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:28 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:28.894 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:28 smithi079 ceph-mon[23699]: pgmap v78: 129 pgs: 129 active+clean; 581 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-09-21T12:30:28.894 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:28 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:28.894 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:28 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:28.894 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:28 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:28.895 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:28 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:28.895 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:28 smithi079 ceph-mon[30023]: pgmap v78: 129 pgs: 129 active+clean; 581 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-09-21T12:30:28.895 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:28 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:29.135 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:28 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:29.135 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:28 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:29.135 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:28 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:29.135 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:28 smithi104 ceph-mon[25639]: pgmap v78: 129 pgs: 129 active+clean; 581 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-09-21T12:30:29.136 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:28 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:29.442 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:30:29 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: [21/Sep/2024:12:30:29] ENGINE Bus STOPPING 2024-09-21T12:30:29.942 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:30:29 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: [21/Sep/2024:12:30:29] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-21T12:30:29.942 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:30:29 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: [21/Sep/2024:12:30:29] ENGINE Bus STOPPED 2024-09-21T12:30:29.942 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:30:29 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: [21/Sep/2024:12:30:29] ENGINE Bus STARTING 2024-09-21T12:30:29.942 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:30:29 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: [21/Sep/2024:12:30:29] ENGINE Serving on http://:::9283 2024-09-21T12:30:29.942 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:30:29 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: [21/Sep/2024:12:30:29] ENGINE Bus STARTED 2024-09-21T12:30:29.943 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:30:29 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: [21/Sep/2024:12:30:29] ENGINE Bus STOPPING 2024-09-21T12:30:29.943 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:29 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:29.943 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:29 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:29.943 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:29 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:29.943 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:29 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:29.943 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:29 smithi079 ceph-mon[23699]: Saving service rgw.foo spec with placement count:2 2024-09-21T12:30:29.943 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:29 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:29.943 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:29 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:29.943 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:29 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:30:29.943 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:29 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:30:29.943 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:29 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:30:29.944 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:29 smithi079 ceph-mon[23699]: pgmap v79: 129 pgs: 129 active+clean; 581 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.1 KiB/s wr, 3 op/s 2024-09-21T12:30:29.944 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:29 smithi079 ceph-mon[23699]: Checking dashboard <-> RGW credentials 2024-09-21T12:30:29.944 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:29 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:29.944 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:29 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:29.944 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:29 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:29.944 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:29 smithi079 ceph-mon[23699]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-21T12:30:29.944 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:29 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:30:29.944 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:29 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:30:29.944 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:29 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:30:29.944 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:29 smithi079 ceph-mon[23699]: pgmap v80: 129 pgs: 129 active+clean; 585 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 129 KiB/s rd, 5.2 KiB/s wr, 244 op/s 2024-09-21T12:30:29.944 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:29 smithi079 ceph-mon[23699]: pgmap v81: 129 pgs: 129 active+clean; 585 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 158 KiB/s rd, 6.3 KiB/s wr, 298 op/s 2024-09-21T12:30:29.944 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:29 smithi079 ceph-mon[23699]: Checking dashboard <-> RGW credentials 2024-09-21T12:30:29.944 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:29 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:29.944 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:29 smithi079 ceph-mon[23699]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-21T12:30:29.945 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:29 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:29.945 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:29 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:29.945 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:29 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:29.945 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:29 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:29.945 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:29 smithi079 ceph-mon[30023]: Saving service rgw.foo spec with placement count:2 2024-09-21T12:30:29.945 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:29 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:29.945 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:29 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:29.945 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:29 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:30:29.945 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:29 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:30:29.945 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:29 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:30:29.945 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:29 smithi079 ceph-mon[30023]: pgmap v79: 129 pgs: 129 active+clean; 581 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.1 KiB/s wr, 3 op/s 2024-09-21T12:30:29.945 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:29 smithi079 ceph-mon[30023]: Checking dashboard <-> RGW credentials 2024-09-21T12:30:29.945 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:29 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:29.945 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:29 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:29.946 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:29 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:29.946 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:29 smithi079 ceph-mon[30023]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-21T12:30:29.946 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:29 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:30:29.946 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:29 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:30:29.946 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:29 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:30:29.946 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:29 smithi079 ceph-mon[30023]: pgmap v80: 129 pgs: 129 active+clean; 585 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 129 KiB/s rd, 5.2 KiB/s wr, 244 op/s 2024-09-21T12:30:29.946 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:29 smithi079 ceph-mon[30023]: pgmap v81: 129 pgs: 129 active+clean; 585 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 158 KiB/s rd, 6.3 KiB/s wr, 298 op/s 2024-09-21T12:30:29.946 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:29 smithi079 ceph-mon[30023]: Checking dashboard <-> RGW credentials 2024-09-21T12:30:29.946 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:29 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:29.946 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:29 smithi079 ceph-mon[30023]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-21T12:30:30.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:29 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:30.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:29 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:30.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:29 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:30.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:29 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:30.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:29 smithi104 ceph-mon[25639]: Saving service rgw.foo spec with placement count:2 2024-09-21T12:30:30.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:29 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:30.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:29 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:30.178 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:29 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:30:30.178 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:29 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:30:30.178 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:29 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:30:30.178 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:29 smithi104 ceph-mon[25639]: pgmap v79: 129 pgs: 129 active+clean; 581 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.1 KiB/s wr, 3 op/s 2024-09-21T12:30:30.178 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:29 smithi104 ceph-mon[25639]: Checking dashboard <-> RGW credentials 2024-09-21T12:30:30.178 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:29 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:30.178 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:29 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:30.178 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:29 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:30.178 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:29 smithi104 ceph-mon[25639]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-21T12:30:30.178 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:29 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:30:30.178 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:29 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:30:30.178 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:29 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:30:30.178 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:29 smithi104 ceph-mon[25639]: pgmap v80: 129 pgs: 129 active+clean; 585 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 129 KiB/s rd, 5.2 KiB/s wr, 244 op/s 2024-09-21T12:30:30.178 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:29 smithi104 ceph-mon[25639]: pgmap v81: 129 pgs: 129 active+clean; 585 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 158 KiB/s rd, 6.3 KiB/s wr, 298 op/s 2024-09-21T12:30:30.178 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:29 smithi104 ceph-mon[25639]: Checking dashboard <-> RGW credentials 2024-09-21T12:30:30.179 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:29 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:30.179 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:29 smithi104 ceph-mon[25639]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-21T12:30:30.324 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:30:30 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: [21/Sep/2024:12:30:30] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-21T12:30:30.324 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:30:30 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: [21/Sep/2024:12:30:30] ENGINE Bus STOPPED 2024-09-21T12:30:30.324 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:30:30 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: [21/Sep/2024:12:30:30] ENGINE Bus STARTING 2024-09-21T12:30:30.583 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:30:30 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: [21/Sep/2024:12:30:30] ENGINE Serving on http://:::9283 2024-09-21T12:30:30.583 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:30:30 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: [21/Sep/2024:12:30:30] ENGINE Bus STARTED 2024-09-21T12:30:30.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:30 smithi104 ceph-mon[25639]: Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON) 2024-09-21T12:30:30.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:30 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:31.035 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:30 smithi079 ceph-mon[23699]: Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON) 2024-09-21T12:30:31.035 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:30 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:31.035 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:30 smithi079 ceph-mon[30023]: Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON) 2024-09-21T12:30:31.035 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:30 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:32.112 INFO:teuthology.orchestra.run.smithi079.stderr:pool 'foo' created 2024-09-21T12:30:32.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:32 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:32.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:32 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:32.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:32 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:30:32.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:32 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:32.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:32 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:30:32.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:32 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:30:32.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:32 smithi079 ceph-mon[30023]: pgmap v82: 129 pgs: 129 active+clean; 585 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 164 KiB/s rd, 6.6 KiB/s wr, 309 op/s 2024-09-21T12:30:32.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:32 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:32.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:32 smithi079 ceph-mon[30023]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-21T12:30:32.193 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:32 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:30:32.193 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:32 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:30:32.193 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:32 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:30:32.193 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:32 smithi079 ceph-mon[30023]: Checking dashboard <-> RGW credentials 2024-09-21T12:30:32.193 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:32 smithi079 ceph-mon[30023]: from='client.? 172.21.15.79:0/193970665' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-09-21T12:30:32.193 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:32 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:32.193 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:32 smithi079 ceph-mon[30023]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-21T12:30:32.193 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:32 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:32.193 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:32 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:32.193 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:32 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:30:32.193 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:32 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:32.193 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:32 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:30:32.194 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:32 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:30:32.194 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:32 smithi079 ceph-mon[23699]: pgmap v82: 129 pgs: 129 active+clean; 585 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 164 KiB/s rd, 6.6 KiB/s wr, 309 op/s 2024-09-21T12:30:32.194 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:32 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:32.194 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:32 smithi079 ceph-mon[23699]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-21T12:30:32.194 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:32 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:30:32.194 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:32 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:30:32.194 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:32 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:30:32.194 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:32 smithi079 ceph-mon[23699]: Checking dashboard <-> RGW credentials 2024-09-21T12:30:32.194 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:32 smithi079 ceph-mon[23699]: from='client.? 172.21.15.79:0/193970665' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-09-21T12:30:32.194 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:32 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:32.194 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:32 smithi079 ceph-mon[23699]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-21T12:30:32.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:32 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:32.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:32 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:32.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:32 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:30:32.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:32 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:32.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:32 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:30:32.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:32 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:30:32.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:32 smithi104 ceph-mon[25639]: pgmap v82: 129 pgs: 129 active+clean; 585 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 164 KiB/s rd, 6.6 KiB/s wr, 309 op/s 2024-09-21T12:30:32.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:32 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:32.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:32 smithi104 ceph-mon[25639]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-21T12:30:32.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:32 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:30:32.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:32 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:30:32.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:32 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:30:32.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:32 smithi104 ceph-mon[25639]: Checking dashboard <-> RGW credentials 2024-09-21T12:30:32.428 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:32 smithi104 ceph-mon[25639]: from='client.? 172.21.15.79:0/193970665' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-09-21T12:30:32.428 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:32 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:32.428 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:32 smithi104 ceph-mon[25639]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-21T12:30:33.333 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:30:33 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: ::ffff:172.21.15.104 - - [21/Sep/2024:12:30:33] "GET /metrics HTTP/1.1" 200 35569 "" "Prometheus/2.43.0" 2024-09-21T12:30:33.334 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:33 smithi079 ceph-mon[23699]: Health check cleared: CEPHADM_STRAY_DAEMON (was: 1 stray daemon(s) not managed by cephadm) 2024-09-21T12:30:33.334 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:33 smithi079 ceph-mon[23699]: Cluster is now healthy 2024-09-21T12:30:33.334 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:33 smithi079 ceph-mon[23699]: from='client.? 172.21.15.79:0/193970665' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "foo"}]': finished 2024-09-21T12:30:33.334 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:33 smithi079 ceph-mon[23699]: osdmap e70: 8 total, 8 up, 8 in 2024-09-21T12:30:33.334 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:33 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:33.334 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:33 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:33.334 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:33 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:33.334 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:33 smithi079 ceph-mon[30023]: Health check cleared: CEPHADM_STRAY_DAEMON (was: 1 stray daemon(s) not managed by cephadm) 2024-09-21T12:30:33.334 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:33 smithi079 ceph-mon[30023]: Cluster is now healthy 2024-09-21T12:30:33.334 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:33 smithi079 ceph-mon[30023]: from='client.? 172.21.15.79:0/193970665' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "foo"}]': finished 2024-09-21T12:30:33.334 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:33 smithi079 ceph-mon[30023]: osdmap e70: 8 total, 8 up, 8 in 2024-09-21T12:30:33.335 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:33 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:33.335 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:33 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:33.335 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:33 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:33.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:33 smithi104 ceph-mon[25639]: Health check cleared: CEPHADM_STRAY_DAEMON (was: 1 stray daemon(s) not managed by cephadm) 2024-09-21T12:30:33.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:33 smithi104 ceph-mon[25639]: Cluster is now healthy 2024-09-21T12:30:33.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:33 smithi104 ceph-mon[25639]: from='client.? 172.21.15.79:0/193970665' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "foo"}]': finished 2024-09-21T12:30:33.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:33 smithi104 ceph-mon[25639]: osdmap e70: 8 total, 8 up, 8 in 2024-09-21T12:30:33.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:33 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:33.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:33 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:33.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:33 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:33.514 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 33168022-7814-11ef-baf6-efdc52797490 -e sha1=441784946648ff53dd310293f9dd752afeeea72a -- bash -c 'rbd pool init foo' 2024-09-21T12:30:34.151 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:30:34.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:34 smithi104 ceph-mon[25639]: pgmap v84: 161 pgs: 32 unknown, 129 active+clean; 585 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 227 KiB/s rd, 7.2 KiB/s wr, 427 op/s 2024-09-21T12:30:34.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:34 smithi104 ceph-mon[25639]: osdmap e71: 8 total, 8 up, 8 in 2024-09-21T12:30:34.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:34 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:34.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:34 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:34.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:34 smithi104 ceph-mon[25639]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T12:30:34.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:34 smithi079 ceph-mon[23699]: pgmap v84: 161 pgs: 32 unknown, 129 active+clean; 585 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 227 KiB/s rd, 7.2 KiB/s wr, 427 op/s 2024-09-21T12:30:34.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:34 smithi079 ceph-mon[23699]: osdmap e71: 8 total, 8 up, 8 in 2024-09-21T12:30:34.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:34 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:34.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:34 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:34.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:34 smithi079 ceph-mon[23699]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T12:30:34.442 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:34 smithi079 ceph-mon[30023]: pgmap v84: 161 pgs: 32 unknown, 129 active+clean; 585 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 227 KiB/s rd, 7.2 KiB/s wr, 427 op/s 2024-09-21T12:30:34.442 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:34 smithi079 ceph-mon[30023]: osdmap e71: 8 total, 8 up, 8 in 2024-09-21T12:30:34.442 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:34 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:34.442 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:34 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:34.442 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:34 smithi079 ceph-mon[30023]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T12:30:35.311 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:35 smithi079 ceph-mon[23699]: osdmap e72: 8 total, 8 up, 8 in 2024-09-21T12:30:35.311 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:35 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:35.311 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:35 smithi079 ceph-mon[30023]: osdmap e72: 8 total, 8 up, 8 in 2024-09-21T12:30:35.311 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:35 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:35.426 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:35 smithi104 ceph-mon[25639]: osdmap e72: 8 total, 8 up, 8 in 2024-09-21T12:30:35.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:35 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:36.676 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:36 smithi104 ceph-mon[25639]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 5.9 KiB/s rd, 5 op/s 2024-09-21T12:30:36.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:36 smithi104 ceph-mon[25639]: from='client.? 172.21.15.79:0/1746407770' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-09-21T12:30:36.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:36 smithi104 ceph-mon[25639]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-09-21T12:30:36.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:36 smithi079 ceph-mon[23699]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 5.9 KiB/s rd, 5 op/s 2024-09-21T12:30:36.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:36 smithi079 ceph-mon[23699]: from='client.? 172.21.15.79:0/1746407770' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-09-21T12:30:36.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:36 smithi079 ceph-mon[23699]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-09-21T12:30:36.692 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:36 smithi079 ceph-mon[30023]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 5.9 KiB/s rd, 5 op/s 2024-09-21T12:30:36.692 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:36 smithi079 ceph-mon[30023]: from='client.? 172.21.15.79:0/1746407770' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-09-21T12:30:36.693 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:36 smithi079 ceph-mon[30023]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-09-21T12:30:37.676 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:37 smithi104 ceph-mon[25639]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]': finished 2024-09-21T12:30:37.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:37 smithi104 ceph-mon[25639]: osdmap e73: 8 total, 8 up, 8 in 2024-09-21T12:30:37.688 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:37 smithi079 ceph-mon[30023]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]': finished 2024-09-21T12:30:37.688 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:37 smithi079 ceph-mon[30023]: osdmap e73: 8 total, 8 up, 8 in 2024-09-21T12:30:37.689 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:37 smithi079 ceph-mon[23699]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]': finished 2024-09-21T12:30:37.689 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:37 smithi079 ceph-mon[23699]: osdmap e73: 8 total, 8 up, 8 in 2024-09-21T12:30:38.318 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:38 smithi079 ceph-mon[23699]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 6.8 KiB/s rd, 6 op/s 2024-09-21T12:30:38.318 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:38 smithi079 ceph-mon[23699]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T12:30:38.318 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:38 smithi079 ceph-mon[23699]: Cluster is now healthy 2024-09-21T12:30:38.318 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:38 smithi079 ceph-mon[23699]: osdmap e74: 8 total, 8 up, 8 in 2024-09-21T12:30:38.318 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:38 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:38.318 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:38 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:38.318 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:38 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:38.318 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:38 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:30:38.318 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:38 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:38.319 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:38 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:30:38.319 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:38 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:30:38.319 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:38 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:38.319 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:38 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi079.tydkfh", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-21T12:30:38.319 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:38 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi079.tydkfh", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-21T12:30:38.319 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:38 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:38.319 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:38 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:30:38.319 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:38 smithi079 ceph-mon[23699]: Deploying daemon rgw.smpl.smithi079.tydkfh on smithi079 2024-09-21T12:30:38.319 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:38 smithi079 ceph-mon[30023]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 6.8 KiB/s rd, 6 op/s 2024-09-21T12:30:38.319 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:38 smithi079 ceph-mon[30023]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T12:30:38.319 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:38 smithi079 ceph-mon[30023]: Cluster is now healthy 2024-09-21T12:30:38.319 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:38 smithi079 ceph-mon[30023]: osdmap e74: 8 total, 8 up, 8 in 2024-09-21T12:30:38.319 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:38 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:38.320 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:38 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:38.320 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:38 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:38.320 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:38 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:30:38.320 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:38 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:38.320 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:38 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:30:38.320 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:38 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:30:38.320 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:38 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:38.320 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:38 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi079.tydkfh", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-21T12:30:38.320 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:38 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi079.tydkfh", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-21T12:30:38.320 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:38 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:38.320 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:38 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:30:38.320 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:38 smithi079 ceph-mon[30023]: Deploying daemon rgw.smpl.smithi079.tydkfh on smithi079 2024-09-21T12:30:38.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:38 smithi104 ceph-mon[25639]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 6.8 KiB/s rd, 6 op/s 2024-09-21T12:30:38.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:38 smithi104 ceph-mon[25639]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T12:30:38.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:38 smithi104 ceph-mon[25639]: Cluster is now healthy 2024-09-21T12:30:38.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:38 smithi104 ceph-mon[25639]: osdmap e74: 8 total, 8 up, 8 in 2024-09-21T12:30:38.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:38 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:38.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:38 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:38.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:38 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:38.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:38 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:30:38.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:38 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:38.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:38 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:30:38.678 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:38 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:30:38.678 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:38 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:38.678 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:38 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi079.tydkfh", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-21T12:30:38.678 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:38 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi079.tydkfh", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-21T12:30:38.678 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:38 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:38.678 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:38 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:30:38.678 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:38 smithi104 ceph-mon[25639]: Deploying daemon rgw.smpl.smithi079.tydkfh on smithi079 2024-09-21T12:30:39.346 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:39 smithi079 ceph-mon[30023]: osdmap e75: 8 total, 8 up, 8 in 2024-09-21T12:30:39.346 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:39 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [7, 2]}]: dispatch 2024-09-21T12:30:39.346 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:39 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]: dispatch 2024-09-21T12:30:39.346 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:39 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]}]: dispatch 2024-09-21T12:30:39.347 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:39 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 5]}]: dispatch 2024-09-21T12:30:39.347 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:39 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T12:30:39.347 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:39 smithi079 ceph-mon[23699]: osdmap e75: 8 total, 8 up, 8 in 2024-09-21T12:30:39.347 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:39 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [7, 2]}]: dispatch 2024-09-21T12:30:39.347 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:39 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]: dispatch 2024-09-21T12:30:39.347 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:39 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]}]: dispatch 2024-09-21T12:30:39.347 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:39 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 5]}]: dispatch 2024-09-21T12:30:39.347 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:39 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T12:30:39.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:39 smithi104 ceph-mon[25639]: osdmap e75: 8 total, 8 up, 8 in 2024-09-21T12:30:39.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:39 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [7, 2]}]: dispatch 2024-09-21T12:30:39.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:39 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]: dispatch 2024-09-21T12:30:39.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:39 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]}]: dispatch 2024-09-21T12:30:39.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:39 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 5]}]: dispatch 2024-09-21T12:30:39.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:39 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T12:30:40.206 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 33168022-7814-11ef-baf6-efdc52797490 -e sha1=441784946648ff53dd310293f9dd752afeeea72a -- bash -c 'ceph orch apply iscsi foo u p' 2024-09-21T12:30:40.435 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:40 smithi079 ceph-mon[23699]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 6.8 KiB/s rd, 6 op/s 2024-09-21T12:30:40.436 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:40 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [7, 2]}]': finished 2024-09-21T12:30:40.436 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:40 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]': finished 2024-09-21T12:30:40.436 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:40 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]}]': finished 2024-09-21T12:30:40.436 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:40 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 5]}]': finished 2024-09-21T12:30:40.436 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:40 smithi079 ceph-mon[23699]: osdmap e76: 8 total, 8 up, 8 in 2024-09-21T12:30:40.436 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:40 smithi079 ceph-mon[30023]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 6.8 KiB/s rd, 6 op/s 2024-09-21T12:30:40.436 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:40 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [7, 2]}]': finished 2024-09-21T12:30:40.436 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:40 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]': finished 2024-09-21T12:30:40.436 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:40 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]}]': finished 2024-09-21T12:30:40.437 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:40 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 5]}]': finished 2024-09-21T12:30:40.437 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:40 smithi079 ceph-mon[30023]: osdmap e76: 8 total, 8 up, 8 in 2024-09-21T12:30:40.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:40 smithi104 ceph-mon[25639]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 6.8 KiB/s rd, 6 op/s 2024-09-21T12:30:40.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:40 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [7, 2]}]': finished 2024-09-21T12:30:40.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:40 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]': finished 2024-09-21T12:30:40.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:40 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]}]': finished 2024-09-21T12:30:40.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:40 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 5]}]': finished 2024-09-21T12:30:40.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:40 smithi104 ceph-mon[25639]: osdmap e76: 8 total, 8 up, 8 in 2024-09-21T12:30:40.808 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:30:41.545 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:41 smithi079 ceph-mon[30023]: osdmap e77: 8 total, 8 up, 8 in 2024-09-21T12:30:41.545 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:41 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:41.545 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:41 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:41.545 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:41 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:41.545 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:41 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi104.riklnj", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-21T12:30:41.545 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:41 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi104.riklnj", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-21T12:30:41.546 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:41 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:41.546 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:41 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:30:41.546 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:41 smithi079 ceph-mon[23699]: osdmap e77: 8 total, 8 up, 8 in 2024-09-21T12:30:41.546 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:41 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:41.546 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:41 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:41.546 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:41 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:41.546 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:41 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi104.riklnj", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-21T12:30:41.546 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:41 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi104.riklnj", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-21T12:30:41.546 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:41 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:41.546 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:41 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:30:41.585 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:41 smithi104 ceph-mon[25639]: osdmap e77: 8 total, 8 up, 8 in 2024-09-21T12:30:41.585 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:41 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:41.585 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:41 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:41.585 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:41 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:41.585 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:41 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi104.riklnj", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-21T12:30:41.585 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:41 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi104.riklnj", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-21T12:30:41.586 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:41 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:41.586 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:41 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:30:42.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:42 smithi104 ceph-mon[25639]: Deploying daemon rgw.smpl.smithi104.riklnj on smithi104 2024-09-21T12:30:42.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:42 smithi104 ceph-mon[25639]: pgmap v95: 161 pgs: 4 peering, 157 active+clean; 585 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 511 B/s wr, 0 op/s 2024-09-21T12:30:42.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:42 smithi079 ceph-mon[23699]: Deploying daemon rgw.smpl.smithi104.riklnj on smithi104 2024-09-21T12:30:42.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:42 smithi079 ceph-mon[23699]: pgmap v95: 161 pgs: 4 peering, 157 active+clean; 585 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 511 B/s wr, 0 op/s 2024-09-21T12:30:42.442 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:42 smithi079 ceph-mon[30023]: Deploying daemon rgw.smpl.smithi104.riklnj on smithi104 2024-09-21T12:30:42.443 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:42 smithi079 ceph-mon[30023]: pgmap v95: 161 pgs: 4 peering, 157 active+clean; 585 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 511 B/s wr, 0 op/s 2024-09-21T12:30:42.521 INFO:teuthology.orchestra.run.smithi079.stdout:Scheduled iscsi.foo update... 2024-09-21T12:30:43.117 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 33168022-7814-11ef-baf6-efdc52797490 -e sha1=441784946648ff53dd310293f9dd752afeeea72a -- bash -c 'sleep 120' 2024-09-21T12:30:43.330 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:30:43.387 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:30:43 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: ::ffff:172.21.15.104 - - [21/Sep/2024:12:30:43] "GET /metrics HTTP/1.1" 200 38178 "" "Prometheus/2.43.0" 2024-09-21T12:30:43.662 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:43 smithi079 ceph-mon[23699]: from='client.24809 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "foo", "api_user": "u", "api_password": "p", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:30:43.663 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:43 smithi079 ceph-mon[23699]: Saving service iscsi.foo spec with placement count:1 2024-09-21T12:30:43.663 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:43 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:43.663 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:43 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:43.663 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:43 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:43.663 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:43 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:43.663 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:43 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:43.663 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:43 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:43.663 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:43 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:43.663 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:43 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:30:43.663 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:43 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:30:43.663 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:43 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:30:43.663 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:43 smithi079 ceph-mon[30023]: from='client.24809 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "foo", "api_user": "u", "api_password": "p", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:30:43.663 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:43 smithi079 ceph-mon[30023]: Saving service iscsi.foo spec with placement count:1 2024-09-21T12:30:43.664 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:43 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:43.664 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:43 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:43.664 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:43 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:43.664 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:43 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:43.664 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:43 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:43.664 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:43 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:43.664 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:43 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:43.664 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:43 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:30:43.664 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:43 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:30:43.664 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:43 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:30:43.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:43 smithi104 ceph-mon[25639]: from='client.24809 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "foo", "api_user": "u", "api_password": "p", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:30:43.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:43 smithi104 ceph-mon[25639]: Saving service iscsi.foo spec with placement count:1 2024-09-21T12:30:43.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:43 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:43.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:43 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:43.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:43 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:43.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:43 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:43.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:43 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:43.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:43 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:43.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:43 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:43.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:43 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:30:43.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:43 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:30:43.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:43 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:30:44.629 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:44 smithi104 ceph-mon[25639]: pgmap v96: 161 pgs: 4 peering, 157 active+clean; 585 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 348 B/s wr, 0 op/s 2024-09-21T12:30:44.630 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:44 smithi104 ceph-mon[25639]: Saving service rgw.smpl spec with placement count:2 2024-09-21T12:30:44.630 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:44 smithi104 ceph-mon[25639]: Checking dashboard <-> RGW credentials 2024-09-21T12:30:44.630 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:44 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:44.630 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:44 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:30:44.630 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:44 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:30:44.630 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:44 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:30:44.630 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:44 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:44.738 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:44 smithi079 ceph-mon[23699]: pgmap v96: 161 pgs: 4 peering, 157 active+clean; 585 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 348 B/s wr, 0 op/s 2024-09-21T12:30:44.738 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:44 smithi079 ceph-mon[23699]: Saving service rgw.smpl spec with placement count:2 2024-09-21T12:30:44.738 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:44 smithi079 ceph-mon[23699]: Checking dashboard <-> RGW credentials 2024-09-21T12:30:44.738 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:44 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:44.738 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:44 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:30:44.738 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:44 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:30:44.739 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:44 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:30:44.739 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:44 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:44.739 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:44 smithi079 ceph-mon[30023]: pgmap v96: 161 pgs: 4 peering, 157 active+clean; 585 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 348 B/s wr, 0 op/s 2024-09-21T12:30:44.739 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:44 smithi079 ceph-mon[30023]: Saving service rgw.smpl spec with placement count:2 2024-09-21T12:30:44.739 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:44 smithi079 ceph-mon[30023]: Checking dashboard <-> RGW credentials 2024-09-21T12:30:44.739 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:44 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:44.739 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:44 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:30:44.739 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:44 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:30:44.739 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:44 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:30:44.739 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:44 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:45.576 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:45 smithi104 ceph-mon[25639]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-21T12:30:45.577 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:45 smithi104 ceph-mon[25639]: Checking dashboard <-> RGW credentials 2024-09-21T12:30:45.577 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:45 smithi104 ceph-mon[25639]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-21T12:30:45.628 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:45 smithi079 ceph-mon[23699]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-21T12:30:45.628 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:45 smithi079 ceph-mon[23699]: Checking dashboard <-> RGW credentials 2024-09-21T12:30:45.629 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:45 smithi079 ceph-mon[23699]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-21T12:30:45.629 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:45 smithi079 ceph-mon[30023]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-21T12:30:45.629 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:45 smithi079 ceph-mon[30023]: Checking dashboard <-> RGW credentials 2024-09-21T12:30:45.629 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:45 smithi079 ceph-mon[30023]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-21T12:30:46.862 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:46 smithi079 ceph-mon[23699]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 193 KiB/s rd, 596 B/s wr, 295 op/s 2024-09-21T12:30:46.862 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:46 smithi079 ceph-mon[30023]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 193 KiB/s rd, 596 B/s wr, 295 op/s 2024-09-21T12:30:46.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:46 smithi104 ceph-mon[25639]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 193 KiB/s rd, 596 B/s wr, 295 op/s 2024-09-21T12:30:48.176 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:47 smithi104 ceph-mon[25639]: osdmap e78: 8 total, 8 up, 8 in 2024-09-21T12:30:48.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:47 smithi104 ceph-mon[25639]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 169 KiB/s rd, 521 B/s wr, 258 op/s 2024-09-21T12:30:48.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:47 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:48.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:47 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:48.191 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:47 smithi079 ceph-mon[30023]: osdmap e78: 8 total, 8 up, 8 in 2024-09-21T12:30:48.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:47 smithi079 ceph-mon[30023]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 169 KiB/s rd, 521 B/s wr, 258 op/s 2024-09-21T12:30:48.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:47 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:48.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:47 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:48.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:47 smithi079 ceph-mon[23699]: osdmap e78: 8 total, 8 up, 8 in 2024-09-21T12:30:48.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:47 smithi079 ceph-mon[23699]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 169 KiB/s rd, 521 B/s wr, 258 op/s 2024-09-21T12:30:48.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:47 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:48.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:47 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:50.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:50 smithi104 ceph-mon[25639]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 150 KiB/s rd, 231 B/s wr, 228 op/s 2024-09-21T12:30:50.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:50 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:50.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:50 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:50.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:50 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:50.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:50 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:50.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:50 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:50.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:50 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:30:50.428 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:50 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:50.428 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:50 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:30:50.428 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:50 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:30:50.428 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:50 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:50.428 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:50 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi079.vlhrek", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-21T12:30:50.428 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:50 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi079.vlhrek", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-09-21T12:30:50.428 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:50 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:30:50.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:50 smithi079 ceph-mon[23699]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 150 KiB/s rd, 231 B/s wr, 228 op/s 2024-09-21T12:30:50.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:50 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:50.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:50 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:50.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:50 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:50.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:50 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:50.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:50 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:50.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:50 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:30:50.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:50 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:50.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:50 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:30:50.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:50 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:30:50.443 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:50 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:50.443 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:50 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi079.vlhrek", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-21T12:30:50.443 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:50 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi079.vlhrek", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-09-21T12:30:50.443 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:50 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:30:50.443 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:50 smithi079 ceph-mon[30023]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 150 KiB/s rd, 231 B/s wr, 228 op/s 2024-09-21T12:30:50.443 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:50 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:50.443 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:50 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:50.443 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:50 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:50.443 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:50 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:50.443 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:50 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:50.443 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:50 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:30:50.443 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:50 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:50.443 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:50 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:30:50.444 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:50 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:30:50.444 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:50 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:50.444 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:50 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi079.vlhrek", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-21T12:30:50.444 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:50 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi079.vlhrek", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-09-21T12:30:50.444 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:50 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:30:51.423 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:51 smithi079 ceph-mon[23699]: Deploying daemon iscsi.foo.smithi079.vlhrek on smithi079 2024-09-21T12:30:51.423 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:51 smithi079 ceph-mon[30023]: Deploying daemon iscsi.foo.smithi079.vlhrek on smithi079 2024-09-21T12:30:51.426 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:51 smithi104 ceph-mon[25639]: Deploying daemon iscsi.foo.smithi079.vlhrek on smithi079 2024-09-21T12:30:52.357 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:52 smithi079 ceph-mon[23699]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 132 KiB/s rd, 204 B/s wr, 202 op/s 2024-09-21T12:30:52.357 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:52 smithi079 ceph-mon[30023]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 132 KiB/s rd, 204 B/s wr, 202 op/s 2024-09-21T12:30:52.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:52 smithi104 ceph-mon[25639]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 132 KiB/s rd, 204 B/s wr, 202 op/s 2024-09-21T12:30:53.360 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:30:53 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: [21/Sep/2024:12:30:53] ENGINE Bus STOPPING 2024-09-21T12:30:53.361 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:30:53 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: ::ffff:172.21.15.104 - - [21/Sep/2024:12:30:53] "GET /metrics HTTP/1.1" 200 38178 "" "Prometheus/2.43.0" 2024-09-21T12:30:53.361 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:30:53 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: [21/Sep/2024:12:30:53] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-21T12:30:53.361 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:30:53 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: [21/Sep/2024:12:30:53] ENGINE Bus STOPPED 2024-09-21T12:30:53.361 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:30:53 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: [21/Sep/2024:12:30:53] ENGINE Bus STARTING 2024-09-21T12:30:53.361 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:30:53 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: [21/Sep/2024:12:30:53] ENGINE Serving on http://:::9283 2024-09-21T12:30:53.361 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:30:53 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: [21/Sep/2024:12:30:53] ENGINE Bus STARTED 2024-09-21T12:30:54.135 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:53 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:54.135 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:53 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:54.135 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:53 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:54.135 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:53 smithi079 ceph-mon[30023]: Checking pool "foo" exists for service iscsi.foo 2024-09-21T12:30:54.135 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:53 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:54.135 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:53 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:30:54.135 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:53 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:30:54.135 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:53 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:30:54.135 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:53 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:54.135 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:53 smithi079 ceph-mon[30023]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-21T12:30:54.135 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:53 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-09-21T12:30:54.135 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:53 smithi079 ceph-mon[30023]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-09-21T12:30:54.136 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:53 smithi079 ceph-mon[30023]: Adding iSCSI gateway http://:@172.21.15.79:5000 to Dashboard 2024-09-21T12:30:54.136 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:53 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-09-21T12:30:54.136 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:53 smithi079 ceph-mon[30023]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-09-21T12:30:54.136 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:53 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:54.136 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:53 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi079"}]: dispatch 2024-09-21T12:30:54.136 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:53 smithi079 ceph-mon[30023]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi079"}]: dispatch 2024-09-21T12:30:54.136 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:53 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:54.136 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:53 smithi079 ceph-mon[30023]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 132 KiB/s rd, 204 B/s wr, 202 op/s 2024-09-21T12:30:54.136 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:53 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:30:54.136 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:53 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:30:54.136 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:53 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:30:54.136 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:53 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:54.136 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:53 smithi079 ceph-mon[30023]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-21T12:30:54.136 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:53 smithi079 ceph-mon[30023]: from='client.? 172.21.15.79:0/2745015831' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-21T12:30:54.136 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:53 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T12:30:54.136 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:53 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:54.136 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:53 smithi079 ceph-mon[30023]: from='client.? 172.21.15.79:0/463482378' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:0/4127214407"}]: dispatch 2024-09-21T12:30:54.137 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:53 smithi079 ceph-mon[30023]: from='client.? ' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:0/4127214407"}]: dispatch 2024-09-21T12:30:54.137 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:53 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:54.137 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:53 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:54.137 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:53 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:54.137 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:53 smithi079 ceph-mon[23699]: Checking pool "foo" exists for service iscsi.foo 2024-09-21T12:30:54.137 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:53 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:54.137 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:53 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:30:54.137 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:53 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:30:54.137 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:53 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:30:54.137 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:53 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:54.137 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:53 smithi079 ceph-mon[23699]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-21T12:30:54.138 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:53 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-09-21T12:30:54.138 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:53 smithi079 ceph-mon[23699]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-09-21T12:30:54.138 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:53 smithi079 ceph-mon[23699]: Adding iSCSI gateway http://:@172.21.15.79:5000 to Dashboard 2024-09-21T12:30:54.138 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:53 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-09-21T12:30:54.138 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:53 smithi079 ceph-mon[23699]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-09-21T12:30:54.138 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:53 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:54.138 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:53 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi079"}]: dispatch 2024-09-21T12:30:54.138 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:53 smithi079 ceph-mon[23699]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi079"}]: dispatch 2024-09-21T12:30:54.138 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:53 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:54.138 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:53 smithi079 ceph-mon[23699]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 132 KiB/s rd, 204 B/s wr, 202 op/s 2024-09-21T12:30:54.138 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:53 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:30:54.138 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:53 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:30:54.138 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:53 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:30:54.138 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:53 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:54.138 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:53 smithi079 ceph-mon[23699]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-21T12:30:54.138 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:53 smithi079 ceph-mon[23699]: from='client.? 172.21.15.79:0/2745015831' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-21T12:30:54.139 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:53 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T12:30:54.139 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:53 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:54.139 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:53 smithi079 ceph-mon[23699]: from='client.? 172.21.15.79:0/463482378' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:0/4127214407"}]: dispatch 2024-09-21T12:30:54.139 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:53 smithi079 ceph-mon[23699]: from='client.? ' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:0/4127214407"}]: dispatch 2024-09-21T12:30:54.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:53 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:54.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:53 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:54.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:53 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:54.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:53 smithi104 ceph-mon[25639]: Checking pool "foo" exists for service iscsi.foo 2024-09-21T12:30:54.178 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:53 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:54.178 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:53 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:30:54.178 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:53 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:30:54.178 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:53 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:30:54.178 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:53 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:54.178 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:53 smithi104 ceph-mon[25639]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-21T12:30:54.178 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:53 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-09-21T12:30:54.178 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:53 smithi104 ceph-mon[25639]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-09-21T12:30:54.178 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:53 smithi104 ceph-mon[25639]: Adding iSCSI gateway http://:@172.21.15.79:5000 to Dashboard 2024-09-21T12:30:54.178 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:53 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-09-21T12:30:54.178 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:53 smithi104 ceph-mon[25639]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-09-21T12:30:54.178 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:53 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:54.178 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:53 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi079"}]: dispatch 2024-09-21T12:30:54.179 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:53 smithi104 ceph-mon[25639]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi079"}]: dispatch 2024-09-21T12:30:54.179 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:53 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:54.179 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:53 smithi104 ceph-mon[25639]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 132 KiB/s rd, 204 B/s wr, 202 op/s 2024-09-21T12:30:54.179 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:53 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:30:54.179 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:53 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:30:54.179 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:53 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:30:54.179 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:53 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:54.179 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:53 smithi104 ceph-mon[25639]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-21T12:30:54.179 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:53 smithi104 ceph-mon[25639]: from='client.? 172.21.15.79:0/2745015831' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-21T12:30:54.179 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:53 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T12:30:54.179 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:53 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:54.179 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:53 smithi104 ceph-mon[25639]: from='client.? 172.21.15.79:0/463482378' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:0/4127214407"}]: dispatch 2024-09-21T12:30:54.179 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:53 smithi104 ceph-mon[25639]: from='client.? ' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:0/4127214407"}]: dispatch 2024-09-21T12:30:55.604 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:55 smithi079 ceph-mon[23699]: from='client.? ' entity='client.iscsi.foo.smithi079.vlhrek' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:0/4127214407"}]': finished 2024-09-21T12:30:55.605 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:55 smithi079 ceph-mon[23699]: osdmap e79: 8 total, 8 up, 8 in 2024-09-21T12:30:55.605 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:55 smithi079 ceph-mon[23699]: from='client.? 172.21.15.79:0/3316837635' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:0/3564175523"}]: dispatch 2024-09-21T12:30:55.605 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:55 smithi079 ceph-mon[23699]: from='client.? ' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:0/3564175523"}]: dispatch 2024-09-21T12:30:55.605 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:55 smithi079 ceph-mon[23699]: mgrmap e21: y(active, since 2m), standbys: x 2024-09-21T12:30:55.605 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:55 smithi079 ceph-mon[30023]: from='client.? ' entity='client.iscsi.foo.smithi079.vlhrek' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:0/4127214407"}]': finished 2024-09-21T12:30:55.605 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:55 smithi079 ceph-mon[30023]: osdmap e79: 8 total, 8 up, 8 in 2024-09-21T12:30:55.605 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:55 smithi079 ceph-mon[30023]: from='client.? 172.21.15.79:0/3316837635' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:0/3564175523"}]: dispatch 2024-09-21T12:30:55.605 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:55 smithi079 ceph-mon[30023]: from='client.? ' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:0/3564175523"}]: dispatch 2024-09-21T12:30:55.605 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:55 smithi079 ceph-mon[30023]: mgrmap e21: y(active, since 2m), standbys: x 2024-09-21T12:30:55.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:55 smithi104 ceph-mon[25639]: from='client.? ' entity='client.iscsi.foo.smithi079.vlhrek' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:0/4127214407"}]': finished 2024-09-21T12:30:55.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:55 smithi104 ceph-mon[25639]: osdmap e79: 8 total, 8 up, 8 in 2024-09-21T12:30:55.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:55 smithi104 ceph-mon[25639]: from='client.? 172.21.15.79:0/3316837635' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:0/3564175523"}]: dispatch 2024-09-21T12:30:55.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:55 smithi104 ceph-mon[25639]: from='client.? ' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:0/3564175523"}]: dispatch 2024-09-21T12:30:55.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:55 smithi104 ceph-mon[25639]: mgrmap e21: y(active, since 2m), standbys: x 2024-09-21T12:30:56.617 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:56 smithi079 ceph-mon[23699]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 245 B/s rd, 122 B/s wr, 1 op/s 2024-09-21T12:30:56.617 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:56 smithi079 ceph-mon[23699]: from='client.? ' entity='client.iscsi.foo.smithi079.vlhrek' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:0/3564175523"}]': finished 2024-09-21T12:30:56.617 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:56 smithi079 ceph-mon[23699]: osdmap e80: 8 total, 8 up, 8 in 2024-09-21T12:30:56.617 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:56 smithi079 ceph-mon[23699]: from='client.? 172.21.15.79:0/989282558' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:0/1473746868"}]: dispatch 2024-09-21T12:30:56.617 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:56 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:56.617 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:56 smithi079 ceph-mon[30023]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 245 B/s rd, 122 B/s wr, 1 op/s 2024-09-21T12:30:56.617 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:56 smithi079 ceph-mon[30023]: from='client.? ' entity='client.iscsi.foo.smithi079.vlhrek' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:0/3564175523"}]': finished 2024-09-21T12:30:56.617 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:56 smithi079 ceph-mon[30023]: osdmap e80: 8 total, 8 up, 8 in 2024-09-21T12:30:56.618 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:56 smithi079 ceph-mon[30023]: from='client.? 172.21.15.79:0/989282558' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:0/1473746868"}]: dispatch 2024-09-21T12:30:56.618 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:56 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:56.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:56 smithi104 ceph-mon[25639]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 245 B/s rd, 122 B/s wr, 1 op/s 2024-09-21T12:30:56.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:56 smithi104 ceph-mon[25639]: from='client.? ' entity='client.iscsi.foo.smithi079.vlhrek' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:0/3564175523"}]': finished 2024-09-21T12:30:56.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:56 smithi104 ceph-mon[25639]: osdmap e80: 8 total, 8 up, 8 in 2024-09-21T12:30:56.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:56 smithi104 ceph-mon[25639]: from='client.? 172.21.15.79:0/989282558' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:0/1473746868"}]: dispatch 2024-09-21T12:30:56.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:56 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:30:57.762 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:57 smithi079 ceph-mon[23699]: from='client.? 172.21.15.79:0/989282558' entity='client.iscsi.foo.smithi079.vlhrek' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:0/1473746868"}]': finished 2024-09-21T12:30:57.762 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:57 smithi079 ceph-mon[23699]: osdmap e81: 8 total, 8 up, 8 in 2024-09-21T12:30:57.763 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:57 smithi079 ceph-mon[23699]: from='client.? 172.21.15.79:0/1360196432' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:0/3967018586"}]: dispatch 2024-09-21T12:30:57.763 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:57 smithi079 ceph-mon[23699]: from='client.? ' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:0/3967018586"}]: dispatch 2024-09-21T12:30:57.763 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:57 smithi079 ceph-mon[30023]: from='client.? 172.21.15.79:0/989282558' entity='client.iscsi.foo.smithi079.vlhrek' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:0/1473746868"}]': finished 2024-09-21T12:30:57.763 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:57 smithi079 ceph-mon[30023]: osdmap e81: 8 total, 8 up, 8 in 2024-09-21T12:30:57.763 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:57 smithi079 ceph-mon[30023]: from='client.? 172.21.15.79:0/1360196432' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:0/3967018586"}]: dispatch 2024-09-21T12:30:57.763 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:57 smithi079 ceph-mon[30023]: from='client.? ' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:0/3967018586"}]: dispatch 2024-09-21T12:30:57.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:57 smithi104 ceph-mon[25639]: from='client.? 172.21.15.79:0/989282558' entity='client.iscsi.foo.smithi079.vlhrek' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:0/1473746868"}]': finished 2024-09-21T12:30:57.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:57 smithi104 ceph-mon[25639]: osdmap e81: 8 total, 8 up, 8 in 2024-09-21T12:30:57.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:57 smithi104 ceph-mon[25639]: from='client.? 172.21.15.79:0/1360196432' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:0/3967018586"}]: dispatch 2024-09-21T12:30:57.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:57 smithi104 ceph-mon[25639]: from='client.? ' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:0/3967018586"}]: dispatch 2024-09-21T12:30:58.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:58 smithi104 ceph-mon[25639]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 170 B/s wr, 1 op/s 2024-09-21T12:30:58.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:58 smithi104 ceph-mon[25639]: from='client.? ' entity='client.iscsi.foo.smithi079.vlhrek' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:0/3967018586"}]': finished 2024-09-21T12:30:58.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:58 smithi104 ceph-mon[25639]: osdmap e82: 8 total, 8 up, 8 in 2024-09-21T12:30:58.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:58 smithi104 ceph-mon[25639]: from='client.? 172.21.15.79:0/60436028' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:6800/306254740"}]: dispatch 2024-09-21T12:30:58.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:58 smithi079 ceph-mon[23699]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 170 B/s wr, 1 op/s 2024-09-21T12:30:58.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:58 smithi079 ceph-mon[23699]: from='client.? ' entity='client.iscsi.foo.smithi079.vlhrek' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:0/3967018586"}]': finished 2024-09-21T12:30:58.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:58 smithi079 ceph-mon[23699]: osdmap e82: 8 total, 8 up, 8 in 2024-09-21T12:30:58.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:58 smithi079 ceph-mon[23699]: from='client.? 172.21.15.79:0/60436028' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:6800/306254740"}]: dispatch 2024-09-21T12:30:58.942 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:58 smithi079 ceph-mon[30023]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 170 B/s wr, 1 op/s 2024-09-21T12:30:58.942 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:58 smithi079 ceph-mon[30023]: from='client.? ' entity='client.iscsi.foo.smithi079.vlhrek' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:0/3967018586"}]': finished 2024-09-21T12:30:58.942 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:58 smithi079 ceph-mon[30023]: osdmap e82: 8 total, 8 up, 8 in 2024-09-21T12:30:58.942 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:58 smithi079 ceph-mon[30023]: from='client.? 172.21.15.79:0/60436028' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:6800/306254740"}]: dispatch 2024-09-21T12:30:59.692 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:59 smithi079 ceph-mon[30023]: from='client.? 172.21.15.79:0/60436028' entity='client.iscsi.foo.smithi079.vlhrek' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:6800/306254740"}]': finished 2024-09-21T12:30:59.692 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:59 smithi079 ceph-mon[30023]: osdmap e83: 8 total, 8 up, 8 in 2024-09-21T12:30:59.692 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:59 smithi079 ceph-mon[30023]: from='client.? 172.21.15.79:0/543390109' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:0/3136532329"}]: dispatch 2024-09-21T12:30:59.692 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:30:59 smithi079 ceph-mon[30023]: from='client.? ' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:0/3136532329"}]: dispatch 2024-09-21T12:30:59.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:59 smithi079 ceph-mon[23699]: from='client.? 172.21.15.79:0/60436028' entity='client.iscsi.foo.smithi079.vlhrek' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:6800/306254740"}]': finished 2024-09-21T12:30:59.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:59 smithi079 ceph-mon[23699]: osdmap e83: 8 total, 8 up, 8 in 2024-09-21T12:30:59.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:59 smithi079 ceph-mon[23699]: from='client.? 172.21.15.79:0/543390109' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:0/3136532329"}]: dispatch 2024-09-21T12:30:59.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:30:59 smithi079 ceph-mon[23699]: from='client.? ' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:0/3136532329"}]: dispatch 2024-09-21T12:30:59.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:59 smithi104 ceph-mon[25639]: from='client.? 172.21.15.79:0/60436028' entity='client.iscsi.foo.smithi079.vlhrek' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:6800/306254740"}]': finished 2024-09-21T12:30:59.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:59 smithi104 ceph-mon[25639]: osdmap e83: 8 total, 8 up, 8 in 2024-09-21T12:30:59.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:59 smithi104 ceph-mon[25639]: from='client.? 172.21.15.79:0/543390109' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:0/3136532329"}]: dispatch 2024-09-21T12:30:59.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:30:59 smithi104 ceph-mon[25639]: from='client.? ' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:0/3136532329"}]: dispatch 2024-09-21T12:31:00.764 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:31:00 smithi079 ceph-mon[30023]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:31:00.764 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:31:00 smithi079 ceph-mon[30023]: from='client.? ' entity='client.iscsi.foo.smithi079.vlhrek' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:0/3136532329"}]': finished 2024-09-21T12:31:00.764 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:31:00 smithi079 ceph-mon[30023]: osdmap e84: 8 total, 8 up, 8 in 2024-09-21T12:31:00.764 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:31:00 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:31:00.764 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:31:00 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:31:00.765 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:31:00 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:31:00.765 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:31:00 smithi079 ceph-mon[30023]: from='client.? 172.21.15.79:0/2170653618' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:6800/3138719122"}]: dispatch 2024-09-21T12:31:00.765 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:31:00 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:31:00.765 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:31:00 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:31:00.765 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:31:00 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:31:00.765 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:31:00 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:31:00.765 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:31:00 smithi079 ceph-mon[23699]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:31:00.765 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:31:00 smithi079 ceph-mon[23699]: from='client.? ' entity='client.iscsi.foo.smithi079.vlhrek' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:0/3136532329"}]': finished 2024-09-21T12:31:00.765 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:31:00 smithi079 ceph-mon[23699]: osdmap e84: 8 total, 8 up, 8 in 2024-09-21T12:31:00.765 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:31:00 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:31:00.765 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:31:00 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:31:00.765 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:31:00 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:31:00.765 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:31:00 smithi079 ceph-mon[23699]: from='client.? 172.21.15.79:0/2170653618' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:6800/3138719122"}]: dispatch 2024-09-21T12:31:00.766 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:31:00 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:31:00.766 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:31:00 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:31:00.766 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:31:00 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:31:00.766 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:31:00 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:31:00.839 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:31:00 smithi104 ceph-mon[25639]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:31:00.839 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:31:00 smithi104 ceph-mon[25639]: from='client.? ' entity='client.iscsi.foo.smithi079.vlhrek' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:0/3136532329"}]': finished 2024-09-21T12:31:00.839 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:31:00 smithi104 ceph-mon[25639]: osdmap e84: 8 total, 8 up, 8 in 2024-09-21T12:31:00.840 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:31:00 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:31:00.840 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:31:00 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:31:00.840 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:31:00 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:31:00.840 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:31:00 smithi104 ceph-mon[25639]: from='client.? 172.21.15.79:0/2170653618' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:6800/3138719122"}]: dispatch 2024-09-21T12:31:00.840 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:31:00 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:31:00.840 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:31:00 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:31:00.840 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:31:00 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:31:00.840 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:31:00 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:31:02.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:31:01 smithi104 ceph-mon[25639]: from='client.? 172.21.15.79:0/2170653618' entity='client.iscsi.foo.smithi079.vlhrek' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:6800/3138719122"}]': finished 2024-09-21T12:31:02.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:31:01 smithi104 ceph-mon[25639]: osdmap e85: 8 total, 8 up, 8 in 2024-09-21T12:31:02.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:31:01 smithi104 ceph-mon[25639]: from='client.? 172.21.15.79:0/761753383' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:0/1207296335"}]: dispatch 2024-09-21T12:31:02.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:31:01 smithi104 ceph-mon[25639]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:31:02.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:31:01 smithi079 ceph-mon[23699]: from='client.? 172.21.15.79:0/2170653618' entity='client.iscsi.foo.smithi079.vlhrek' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:6800/3138719122"}]': finished 2024-09-21T12:31:02.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:31:01 smithi079 ceph-mon[23699]: osdmap e85: 8 total, 8 up, 8 in 2024-09-21T12:31:02.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:31:01 smithi079 ceph-mon[23699]: from='client.? 172.21.15.79:0/761753383' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:0/1207296335"}]: dispatch 2024-09-21T12:31:02.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:31:01 smithi079 ceph-mon[23699]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:31:02.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:31:01 smithi079 ceph-mon[30023]: from='client.? 172.21.15.79:0/2170653618' entity='client.iscsi.foo.smithi079.vlhrek' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:6800/3138719122"}]': finished 2024-09-21T12:31:02.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:31:01 smithi079 ceph-mon[30023]: osdmap e85: 8 total, 8 up, 8 in 2024-09-21T12:31:02.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:31:01 smithi079 ceph-mon[30023]: from='client.? 172.21.15.79:0/761753383' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:0/1207296335"}]: dispatch 2024-09-21T12:31:02.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:31:01 smithi079 ceph-mon[30023]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:31:03.076 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:31:02 smithi079 ceph-mon[30023]: from='client.? 172.21.15.79:0/761753383' entity='client.iscsi.foo.smithi079.vlhrek' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:0/1207296335"}]': finished 2024-09-21T12:31:03.077 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:31:02 smithi079 ceph-mon[30023]: osdmap e86: 8 total, 8 up, 8 in 2024-09-21T12:31:03.077 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:31:02 smithi079 ceph-mon[30023]: from='client.? 172.21.15.79:0/1107139281' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:0/1561731591"}]: dispatch 2024-09-21T12:31:03.077 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:31:02 smithi079 ceph-mon[23699]: from='client.? 172.21.15.79:0/761753383' entity='client.iscsi.foo.smithi079.vlhrek' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:0/1207296335"}]': finished 2024-09-21T12:31:03.077 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:31:02 smithi079 ceph-mon[23699]: osdmap e86: 8 total, 8 up, 8 in 2024-09-21T12:31:03.077 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:31:02 smithi079 ceph-mon[23699]: from='client.? 172.21.15.79:0/1107139281' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:0/1561731591"}]: dispatch 2024-09-21T12:31:03.176 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:31:02 smithi104 ceph-mon[25639]: from='client.? 172.21.15.79:0/761753383' entity='client.iscsi.foo.smithi079.vlhrek' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:0/1207296335"}]': finished 2024-09-21T12:31:03.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:31:02 smithi104 ceph-mon[25639]: osdmap e86: 8 total, 8 up, 8 in 2024-09-21T12:31:03.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:31:02 smithi104 ceph-mon[25639]: from='client.? 172.21.15.79:0/1107139281' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:0/1561731591"}]: dispatch 2024-09-21T12:31:03.442 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:31:03 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: ::ffff:172.21.15.104 - - [21/Sep/2024:12:31:03] "GET /metrics HTTP/1.1" 200 38500 "" "Prometheus/2.43.0" 2024-09-21T12:31:04.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:31:03 smithi104 ceph-mon[25639]: from='client.? 172.21.15.79:0/1107139281' entity='client.iscsi.foo.smithi079.vlhrek' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:0/1561731591"}]': finished 2024-09-21T12:31:04.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:31:03 smithi104 ceph-mon[25639]: osdmap e87: 8 total, 8 up, 8 in 2024-09-21T12:31:04.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:31:03 smithi104 ceph-mon[25639]: from='client.? 172.21.15.79:0/1929671185' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:0/2089075974"}]: dispatch 2024-09-21T12:31:04.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:31:03 smithi104 ceph-mon[25639]: from='client.15051 -' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:31:04.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:31:03 smithi104 ceph-mon[25639]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:31:04.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:31:03 smithi079 ceph-mon[23699]: from='client.? 172.21.15.79:0/1107139281' entity='client.iscsi.foo.smithi079.vlhrek' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:0/1561731591"}]': finished 2024-09-21T12:31:04.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:31:03 smithi079 ceph-mon[23699]: osdmap e87: 8 total, 8 up, 8 in 2024-09-21T12:31:04.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:31:03 smithi079 ceph-mon[23699]: from='client.? 172.21.15.79:0/1929671185' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:0/2089075974"}]: dispatch 2024-09-21T12:31:04.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:31:03 smithi079 ceph-mon[23699]: from='client.15051 -' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:31:04.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:31:03 smithi079 ceph-mon[23699]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:31:04.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:31:03 smithi079 ceph-mon[30023]: from='client.? 172.21.15.79:0/1107139281' entity='client.iscsi.foo.smithi079.vlhrek' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:0/1561731591"}]': finished 2024-09-21T12:31:04.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:31:03 smithi079 ceph-mon[30023]: osdmap e87: 8 total, 8 up, 8 in 2024-09-21T12:31:04.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:31:03 smithi079 ceph-mon[30023]: from='client.? 172.21.15.79:0/1929671185' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:0/2089075974"}]: dispatch 2024-09-21T12:31:04.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:31:03 smithi079 ceph-mon[30023]: from='client.15051 -' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:31:04.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:31:03 smithi079 ceph-mon[30023]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:31:05.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:31:04 smithi104 ceph-mon[25639]: from='client.? 172.21.15.79:0/1929671185' entity='client.iscsi.foo.smithi079.vlhrek' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:0/2089075974"}]': finished 2024-09-21T12:31:05.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:31:04 smithi104 ceph-mon[25639]: osdmap e88: 8 total, 8 up, 8 in 2024-09-21T12:31:05.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:31:04 smithi104 ceph-mon[25639]: from='client.? 172.21.15.79:0/2988566656' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:6801/3138719122"}]: dispatch 2024-09-21T12:31:05.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:31:04 smithi104 ceph-mon[25639]: from='client.? ' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:6801/3138719122"}]: dispatch 2024-09-21T12:31:05.191 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:31:04 smithi079 ceph-mon[30023]: from='client.? 172.21.15.79:0/1929671185' entity='client.iscsi.foo.smithi079.vlhrek' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:0/2089075974"}]': finished 2024-09-21T12:31:05.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:31:04 smithi079 ceph-mon[30023]: osdmap e88: 8 total, 8 up, 8 in 2024-09-21T12:31:05.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:31:04 smithi079 ceph-mon[30023]: from='client.? 172.21.15.79:0/2988566656' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:6801/3138719122"}]: dispatch 2024-09-21T12:31:05.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:31:04 smithi079 ceph-mon[30023]: from='client.? ' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:6801/3138719122"}]: dispatch 2024-09-21T12:31:05.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:31:04 smithi079 ceph-mon[23699]: from='client.? 172.21.15.79:0/1929671185' entity='client.iscsi.foo.smithi079.vlhrek' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:0/2089075974"}]': finished 2024-09-21T12:31:05.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:31:04 smithi079 ceph-mon[23699]: osdmap e88: 8 total, 8 up, 8 in 2024-09-21T12:31:05.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:31:04 smithi079 ceph-mon[23699]: from='client.? 172.21.15.79:0/2988566656' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:6801/3138719122"}]: dispatch 2024-09-21T12:31:05.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:31:04 smithi079 ceph-mon[23699]: from='client.? ' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:6801/3138719122"}]: dispatch 2024-09-21T12:31:06.110 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:31:05 smithi104 ceph-mon[25639]: from='client.? ' entity='client.iscsi.foo.smithi079.vlhrek' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:6801/3138719122"}]': finished 2024-09-21T12:31:06.110 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:31:05 smithi104 ceph-mon[25639]: osdmap e89: 8 total, 8 up, 8 in 2024-09-21T12:31:06.111 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:31:05 smithi104 ceph-mon[25639]: from='client.? 172.21.15.79:0/2827442012' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:6800/3479175059"}]: dispatch 2024-09-21T12:31:06.111 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:31:05 smithi104 ceph-mon[25639]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:31:06.191 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:31:05 smithi079 ceph-mon[30023]: from='client.? ' entity='client.iscsi.foo.smithi079.vlhrek' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:6801/3138719122"}]': finished 2024-09-21T12:31:06.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:31:05 smithi079 ceph-mon[30023]: osdmap e89: 8 total, 8 up, 8 in 2024-09-21T12:31:06.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:31:05 smithi079 ceph-mon[30023]: from='client.? 172.21.15.79:0/2827442012' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:6800/3479175059"}]: dispatch 2024-09-21T12:31:06.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:31:05 smithi079 ceph-mon[30023]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:31:06.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:31:05 smithi079 ceph-mon[23699]: from='client.? ' entity='client.iscsi.foo.smithi079.vlhrek' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:6801/3138719122"}]': finished 2024-09-21T12:31:06.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:31:05 smithi079 ceph-mon[23699]: osdmap e89: 8 total, 8 up, 8 in 2024-09-21T12:31:06.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:31:05 smithi079 ceph-mon[23699]: from='client.? 172.21.15.79:0/2827442012' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:6800/3479175059"}]: dispatch 2024-09-21T12:31:06.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:31:05 smithi079 ceph-mon[23699]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:31:07.013 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:31:06 smithi104 ceph-mon[25639]: from='client.? 172.21.15.79:0/2827442012' entity='client.iscsi.foo.smithi079.vlhrek' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:6800/3479175059"}]': finished 2024-09-21T12:31:07.013 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:31:06 smithi104 ceph-mon[25639]: osdmap e90: 8 total, 8 up, 8 in 2024-09-21T12:31:07.014 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:31:06 smithi104 ceph-mon[25639]: from='client.? 172.21.15.79:0/1623637527' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:6801/3479175059"}]: dispatch 2024-09-21T12:31:07.014 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:31:06 smithi104 ceph-mon[25639]: from='client.? ' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:6801/3479175059"}]: dispatch 2024-09-21T12:31:07.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:31:06 smithi079 ceph-mon[23699]: from='client.? 172.21.15.79:0/2827442012' entity='client.iscsi.foo.smithi079.vlhrek' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:6800/3479175059"}]': finished 2024-09-21T12:31:07.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:31:06 smithi079 ceph-mon[23699]: osdmap e90: 8 total, 8 up, 8 in 2024-09-21T12:31:07.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:31:06 smithi079 ceph-mon[23699]: from='client.? 172.21.15.79:0/1623637527' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:6801/3479175059"}]: dispatch 2024-09-21T12:31:07.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:31:06 smithi079 ceph-mon[23699]: from='client.? ' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:6801/3479175059"}]: dispatch 2024-09-21T12:31:07.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:31:06 smithi079 ceph-mon[30023]: from='client.? 172.21.15.79:0/2827442012' entity='client.iscsi.foo.smithi079.vlhrek' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:6800/3479175059"}]': finished 2024-09-21T12:31:07.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:31:06 smithi079 ceph-mon[30023]: osdmap e90: 8 total, 8 up, 8 in 2024-09-21T12:31:07.193 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:31:06 smithi079 ceph-mon[30023]: from='client.? 172.21.15.79:0/1623637527' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:6801/3479175059"}]: dispatch 2024-09-21T12:31:07.193 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:31:06 smithi079 ceph-mon[30023]: from='client.? ' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:6801/3479175059"}]: dispatch 2024-09-21T12:31:08.176 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:31:07 smithi104 ceph-mon[25639]: from='client.? ' entity='client.iscsi.foo.smithi079.vlhrek' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:6801/3479175059"}]': finished 2024-09-21T12:31:08.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:31:07 smithi104 ceph-mon[25639]: osdmap e91: 8 total, 8 up, 8 in 2024-09-21T12:31:08.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:31:07 smithi104 ceph-mon[25639]: from='client.? 172.21.15.79:0/4044951930' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:0/2503007178"}]: dispatch 2024-09-21T12:31:08.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:31:07 smithi104 ceph-mon[25639]: from='client.? ' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:0/2503007178"}]: dispatch 2024-09-21T12:31:08.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:31:07 smithi104 ceph-mon[25639]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:31:08.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:31:07 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:31:08.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:31:07 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:31:08.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:31:07 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:31:08.191 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:31:07 smithi079 ceph-mon[23699]: from='client.? ' entity='client.iscsi.foo.smithi079.vlhrek' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:6801/3479175059"}]': finished 2024-09-21T12:31:08.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:31:07 smithi079 ceph-mon[23699]: osdmap e91: 8 total, 8 up, 8 in 2024-09-21T12:31:08.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:31:07 smithi079 ceph-mon[23699]: from='client.? 172.21.15.79:0/4044951930' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:0/2503007178"}]: dispatch 2024-09-21T12:31:08.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:31:07 smithi079 ceph-mon[23699]: from='client.? ' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:0/2503007178"}]: dispatch 2024-09-21T12:31:08.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:31:07 smithi079 ceph-mon[23699]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:31:08.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:31:07 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:31:08.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:31:07 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:31:08.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:31:07 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:31:08.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:31:07 smithi079 ceph-mon[30023]: from='client.? ' entity='client.iscsi.foo.smithi079.vlhrek' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:6801/3479175059"}]': finished 2024-09-21T12:31:08.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:31:07 smithi079 ceph-mon[30023]: osdmap e91: 8 total, 8 up, 8 in 2024-09-21T12:31:08.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:31:07 smithi079 ceph-mon[30023]: from='client.? 172.21.15.79:0/4044951930' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:0/2503007178"}]: dispatch 2024-09-21T12:31:08.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:31:07 smithi079 ceph-mon[30023]: from='client.? ' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:0/2503007178"}]: dispatch 2024-09-21T12:31:08.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:31:07 smithi079 ceph-mon[30023]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:31:08.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:31:07 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:31:08.193 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:31:07 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:31:08.193 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:31:07 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:31:09.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:31:08 smithi104 ceph-mon[25639]: from='client.? ' entity='client.iscsi.foo.smithi079.vlhrek' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:0/2503007178"}]': finished 2024-09-21T12:31:09.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:31:08 smithi104 ceph-mon[25639]: osdmap e92: 8 total, 8 up, 8 in 2024-09-21T12:31:09.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:31:08 smithi104 ceph-mon[25639]: from='client.? 172.21.15.79:0/3087024011' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:6801/306254740"}]: dispatch 2024-09-21T12:31:09.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:31:08 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T12:31:09.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:31:08 smithi079 ceph-mon[23699]: from='client.? ' entity='client.iscsi.foo.smithi079.vlhrek' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:0/2503007178"}]': finished 2024-09-21T12:31:09.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:31:08 smithi079 ceph-mon[23699]: osdmap e92: 8 total, 8 up, 8 in 2024-09-21T12:31:09.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:31:08 smithi079 ceph-mon[23699]: from='client.? 172.21.15.79:0/3087024011' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:6801/306254740"}]: dispatch 2024-09-21T12:31:09.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:31:08 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T12:31:09.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:31:08 smithi079 ceph-mon[30023]: from='client.? ' entity='client.iscsi.foo.smithi079.vlhrek' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:0/2503007178"}]': finished 2024-09-21T12:31:09.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:31:08 smithi079 ceph-mon[30023]: osdmap e92: 8 total, 8 up, 8 in 2024-09-21T12:31:09.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:31:08 smithi079 ceph-mon[30023]: from='client.? 172.21.15.79:0/3087024011' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:6801/306254740"}]: dispatch 2024-09-21T12:31:09.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:31:08 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T12:31:10.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:31:09 smithi104 ceph-mon[25639]: from='client.? 172.21.15.79:0/3087024011' entity='client.iscsi.foo.smithi079.vlhrek' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:6801/306254740"}]': finished 2024-09-21T12:31:10.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:31:09 smithi104 ceph-mon[25639]: osdmap e93: 8 total, 8 up, 8 in 2024-09-21T12:31:10.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:31:09 smithi104 ceph-mon[25639]: from='client.? 172.21.15.79:0/58325568' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:0/1823093535"}]: dispatch 2024-09-21T12:31:10.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:31:09 smithi104 ceph-mon[25639]: from='client.? ' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:0/1823093535"}]: dispatch 2024-09-21T12:31:10.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:31:09 smithi104 ceph-mon[25639]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:31:10.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:31:09 smithi079 ceph-mon[23699]: from='client.? 172.21.15.79:0/3087024011' entity='client.iscsi.foo.smithi079.vlhrek' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:6801/306254740"}]': finished 2024-09-21T12:31:10.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:31:09 smithi079 ceph-mon[23699]: osdmap e93: 8 total, 8 up, 8 in 2024-09-21T12:31:10.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:31:09 smithi079 ceph-mon[23699]: from='client.? 172.21.15.79:0/58325568' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:0/1823093535"}]: dispatch 2024-09-21T12:31:10.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:31:09 smithi079 ceph-mon[23699]: from='client.? ' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:0/1823093535"}]: dispatch 2024-09-21T12:31:10.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:31:09 smithi079 ceph-mon[23699]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:31:10.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:31:09 smithi079 ceph-mon[30023]: from='client.? 172.21.15.79:0/3087024011' entity='client.iscsi.foo.smithi079.vlhrek' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:6801/306254740"}]': finished 2024-09-21T12:31:10.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:31:09 smithi079 ceph-mon[30023]: osdmap e93: 8 total, 8 up, 8 in 2024-09-21T12:31:10.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:31:09 smithi079 ceph-mon[30023]: from='client.? 172.21.15.79:0/58325568' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:0/1823093535"}]: dispatch 2024-09-21T12:31:10.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:31:09 smithi079 ceph-mon[30023]: from='client.? ' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:0/1823093535"}]: dispatch 2024-09-21T12:31:10.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:31:09 smithi079 ceph-mon[30023]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:31:11.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:31:10 smithi104 ceph-mon[25639]: from='client.? ' entity='client.iscsi.foo.smithi079.vlhrek' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:0/1823093535"}]': finished 2024-09-21T12:31:11.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:31:10 smithi104 ceph-mon[25639]: osdmap e94: 8 total, 8 up, 8 in 2024-09-21T12:31:11.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:31:10 smithi079 ceph-mon[23699]: from='client.? ' entity='client.iscsi.foo.smithi079.vlhrek' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:0/1823093535"}]': finished 2024-09-21T12:31:11.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:31:10 smithi079 ceph-mon[23699]: osdmap e94: 8 total, 8 up, 8 in 2024-09-21T12:31:11.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:31:10 smithi079 ceph-mon[30023]: from='client.? ' entity='client.iscsi.foo.smithi079.vlhrek' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:0/1823093535"}]': finished 2024-09-21T12:31:11.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:31:10 smithi079 ceph-mon[30023]: osdmap e94: 8 total, 8 up, 8 in 2024-09-21T12:31:12.093 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:31:11 smithi079 ceph-mon[23699]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:31:12.093 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:31:11 smithi079 ceph-mon[30023]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:31:12.176 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:31:11 smithi104 ceph-mon[25639]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:31:13.406 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:31:13 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: ::ffff:172.21.15.104 - - [21/Sep/2024:12:31:13] "GET /metrics HTTP/1.1" 200 38510 "" "Prometheus/2.43.0" 2024-09-21T12:31:14.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:31:14 smithi104 ceph-mon[25639]: from='client.15051 -' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:31:14.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:31:14 smithi104 ceph-mon[25639]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:31:14.429 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:31:14 smithi079 ceph-mon[23699]: from='client.15051 -' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:31:14.429 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:31:14 smithi079 ceph-mon[23699]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:31:14.429 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:31:14 smithi079 ceph-mon[30023]: from='client.15051 -' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:31:14.429 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:31:14 smithi079 ceph-mon[30023]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:31:16.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:31:16 smithi104 ceph-mon[25639]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-09-21T12:31:16.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:31:16 smithi079 ceph-mon[23699]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-09-21T12:31:16.443 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:31:16 smithi079 ceph-mon[30023]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-09-21T12:31:18.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:31:17 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:31:18.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:31:17 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:31:18.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:31:17 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:31:18.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:31:17 smithi104 ceph-mon[25639]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:31:18.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:31:17 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:31:18.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:31:17 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:31:18.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:31:17 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:31:18.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:31:17 smithi079 ceph-mon[23699]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:31:18.443 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:31:17 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:31:18.443 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:31:17 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:31:18.443 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:31:17 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:31:18.443 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:31:17 smithi079 ceph-mon[30023]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:31:20.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:31:20 smithi104 ceph-mon[25639]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-21T12:31:20.441 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:31:20 smithi079 ceph-mon[23699]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-21T12:31:20.442 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:31:20 smithi079 ceph-mon[30023]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-21T12:31:22.426 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:31:22 smithi104 ceph-mon[25639]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 905 B/s rd, 0 op/s 2024-09-21T12:31:22.442 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:31:22 smithi079 ceph-mon[30023]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 905 B/s rd, 0 op/s 2024-09-21T12:31:22.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:31:22 smithi079 ceph-mon[23699]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 905 B/s rd, 0 op/s 2024-09-21T12:31:23.347 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:31:23 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: ::ffff:172.21.15.104 - - [21/Sep/2024:12:31:23] "GET /metrics HTTP/1.1" 200 38510 "" "Prometheus/2.43.0" 2024-09-21T12:31:24.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:31:24 smithi104 ceph-mon[25639]: from='client.15051 -' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:31:24.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:31:24 smithi104 ceph-mon[25639]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:31:24.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:31:24 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T12:31:24.442 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:31:24 smithi079 ceph-mon[30023]: from='client.15051 -' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:31:24.442 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:31:24 smithi079 ceph-mon[30023]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:31:24.442 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:31:24 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T12:31:24.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:31:24 smithi079 ceph-mon[23699]: from='client.15051 -' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:31:24.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:31:24 smithi079 ceph-mon[23699]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:31:24.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:31:24 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T12:31:26.344 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:31:26 smithi104 ceph-mon[25639]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:31:26.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:31:26 smithi079 ceph-mon[23699]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:31:26.442 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:31:26 smithi079 ceph-mon[30023]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:31:28.656 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:31:28 smithi104 ceph-mon[25639]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:31:28.656 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:31:28 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:31:28.656 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:31:28 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:31:28.656 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:31:28 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:31:28.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:31:28 smithi079 ceph-mon[23699]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:31:28.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:31:28 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:31:28.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:31:28 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:31:28.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:31:28 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:31:28.692 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:31:28 smithi079 ceph-mon[30023]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:31:28.692 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:31:28 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:31:28.692 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:31:28 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:31:28.693 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:31:28 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:31:30.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:31:30 smithi104 ceph-mon[25639]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:31:30.691 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:31:30 smithi079 ceph-mon[23699]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:31:30.692 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:31:30 smithi079 ceph-mon[30023]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:31:32.676 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:31:32 smithi104 ceph-mon[25639]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:31:32.691 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:31:32 smithi079 ceph-mon[23699]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:31:32.692 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:31:32 smithi079 ceph-mon[30023]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:31:33.442 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:31:33 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: ::ffff:172.21.15.104 - - [21/Sep/2024:12:31:33] "GET /metrics HTTP/1.1" 200 38509 "" "Prometheus/2.43.0" 2024-09-21T12:31:34.676 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:31:34 smithi104 ceph-mon[25639]: from='client.15051 -' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:31:34.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:31:34 smithi104 ceph-mon[25639]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:31:34.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:31:34 smithi079 ceph-mon[23699]: from='client.15051 -' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:31:34.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:31:34 smithi079 ceph-mon[23699]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:31:34.692 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:31:34 smithi079 ceph-mon[30023]: from='client.15051 -' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:31:34.692 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:31:34 smithi079 ceph-mon[30023]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:31:36.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:31:36 smithi079 ceph-mon[23699]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:31:36.692 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:31:36 smithi079 ceph-mon[30023]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:31:36.926 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:31:36 smithi104 ceph-mon[25639]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:31:38.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:31:38 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:31:38.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:31:38 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:31:38.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:31:38 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:31:38.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:31:38 smithi104 ceph-mon[25639]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:31:38.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:31:38 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:31:38.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:31:38 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:31:38.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:31:38 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:31:38.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:31:38 smithi079 ceph-mon[23699]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:31:38.443 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:31:38 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:31:38.443 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:31:38 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:31:38.443 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:31:38 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:31:38.443 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:31:38 smithi079 ceph-mon[30023]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:31:39.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:31:39 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T12:31:39.441 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:31:39 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T12:31:39.442 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:31:39 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T12:31:40.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:31:40 smithi104 ceph-mon[25639]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:31:40.442 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:31:40 smithi079 ceph-mon[30023]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:31:40.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:31:40 smithi079 ceph-mon[23699]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:31:42.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:31:42 smithi079 ceph-mon[23699]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:31:42.442 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:31:42 smithi079 ceph-mon[30023]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:31:42.676 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:31:42 smithi104 ceph-mon[25639]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:31:43.442 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:31:43 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: ::ffff:172.21.15.104 - - [21/Sep/2024:12:31:43] "GET /metrics HTTP/1.1" 200 38505 "" "Prometheus/2.43.0" 2024-09-21T12:31:44.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:31:44 smithi079 ceph-mon[23699]: from='client.15051 -' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:31:44.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:31:44 smithi079 ceph-mon[23699]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:31:44.442 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:31:44 smithi079 ceph-mon[30023]: from='client.15051 -' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:31:44.442 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:31:44 smithi079 ceph-mon[30023]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:31:44.676 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:31:44 smithi104 ceph-mon[25639]: from='client.15051 -' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:31:44.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:31:44 smithi104 ceph-mon[25639]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:31:46.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:31:46 smithi079 ceph-mon[23699]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:31:46.442 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:31:46 smithi079 ceph-mon[30023]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:31:46.676 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:31:46 smithi104 ceph-mon[25639]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:31:48.907 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:31:48 smithi104 ceph-mon[25639]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:31:48.907 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:31:48 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:31:48.907 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:31:48 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:31:48.908 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:31:48 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:31:48.942 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:31:48 smithi079 ceph-mon[30023]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:31:48.942 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:31:48 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:31:48.942 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:31:48 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:31:48.942 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:31:48 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:31:48.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:31:48 smithi079 ceph-mon[23699]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:31:48.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:31:48 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:31:48.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:31:48 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:31:48.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:31:48 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:31:50.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:31:50 smithi104 ceph-mon[25639]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:31:50.941 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:31:50 smithi079 ceph-mon[30023]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:31:50.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:31:50 smithi079 ceph-mon[23699]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:31:52.926 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:31:52 smithi104 ceph-mon[25639]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:31:52.941 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:31:52 smithi079 ceph-mon[23699]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:31:52.941 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:31:52 smithi079 ceph-mon[30023]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:31:53.442 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:31:53 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: ::ffff:172.21.15.104 - - [21/Sep/2024:12:31:53] "GET /metrics HTTP/1.1" 200 38505 "" "Prometheus/2.43.0" 2024-09-21T12:31:53.926 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:31:53 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T12:31:53.941 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:31:53 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T12:31:53.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:31:53 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T12:31:54.902 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:31:54 smithi079 ceph-mon[30023]: from='client.15051 -' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:31:54.902 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:31:54 smithi079 ceph-mon[30023]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:31:54.902 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:31:54 smithi079 ceph-mon[23699]: from='client.15051 -' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:31:54.902 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:31:54 smithi079 ceph-mon[23699]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:31:54.926 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:31:54 smithi104 ceph-mon[25639]: from='client.15051 -' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:31:54.926 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:31:54 smithi104 ceph-mon[25639]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:31:56.881 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:31:56 smithi079 ceph-mon[23699]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:31:56.881 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:31:56 smithi079 ceph-mon[30023]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:31:56.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:31:56 smithi104 ceph-mon[25639]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:31:58.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:31:58 smithi104 ceph-mon[25639]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:31:58.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:31:58 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:31:58.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:31:58 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:31:58.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:31:58 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:31:58.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:31:58 smithi079 ceph-mon[23699]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:31:58.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:31:58 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:31:58.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:31:58 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:31:58.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:31:58 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:31:58.692 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:31:58 smithi079 ceph-mon[30023]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:31:58.692 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:31:58 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:31:58.692 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:31:58 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:31:58.692 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:31:58 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:32:00.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:32:00 smithi104 ceph-mon[25639]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:32:00.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:32:00 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:32:00.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:32:00 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:32:00.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:32:00 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:32:00.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:32:00 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:32:00.692 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:32:00 smithi079 ceph-mon[30023]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:32:00.692 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:32:00 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:32:00.692 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:32:00 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:32:00.692 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:32:00 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:32:00.692 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:32:00 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:32:00.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:32:00 smithi079 ceph-mon[23699]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:32:00.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:32:00 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:32:00.693 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:32:00 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:32:00.693 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:32:00 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:32:00.693 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:32:00 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:32:02.676 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:32:02 smithi104 ceph-mon[25639]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:32:02.691 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:32:02 smithi079 ceph-mon[23699]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:32:02.692 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:32:02 smithi079 ceph-mon[30023]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:32:03.442 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:32:03 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: ::ffff:172.21.15.104 - - [21/Sep/2024:12:32:03] "GET /metrics HTTP/1.1" 200 38509 "" "Prometheus/2.43.0" 2024-09-21T12:32:04.678 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:32:04 smithi104 ceph-mon[25639]: from='client.15051 -' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:32:04.678 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:32:04 smithi104 ceph-mon[25639]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:32:04.692 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:32:04 smithi079 ceph-mon[30023]: from='client.15051 -' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:32:04.692 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:32:04 smithi079 ceph-mon[30023]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:32:04.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:32:04 smithi079 ceph-mon[23699]: from='client.15051 -' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:32:04.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:32:04 smithi079 ceph-mon[23699]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:32:06.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:32:06 smithi104 ceph-mon[25639]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:32:06.692 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:32:06 smithi079 ceph-mon[30023]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:32:06.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:32:06 smithi079 ceph-mon[23699]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:32:08.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:32:08 smithi104 ceph-mon[25639]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:32:08.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:32:08 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:32:08.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:32:08 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:32:08.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:32:08 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:32:08.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:32:08 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T12:32:08.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:32:08 smithi079 ceph-mon[23699]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:32:08.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:32:08 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:32:08.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:32:08 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:32:08.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:32:08 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:32:08.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:32:08 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T12:32:08.942 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:32:08 smithi079 ceph-mon[30023]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:32:08.942 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:32:08 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:32:08.942 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:32:08 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:32:08.942 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:32:08 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:32:08.942 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:32:08 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T12:32:10.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:32:10 smithi079 ceph-mon[23699]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:32:10.942 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:32:10 smithi079 ceph-mon[30023]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:32:11.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:32:10 smithi104 ceph-mon[25639]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:32:12.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:32:12 smithi079 ceph-mon[23699]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:32:12.942 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:32:12 smithi079 ceph-mon[30023]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:32:13.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:32:12 smithi104 ceph-mon[25639]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:32:13.442 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:32:13 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: ::ffff:172.21.15.104 - - [21/Sep/2024:12:32:13] "GET /metrics HTTP/1.1" 200 38510 "" "Prometheus/2.43.0" 2024-09-21T12:32:14.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:32:14 smithi079 ceph-mon[23699]: from='client.15051 -' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:32:14.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:32:14 smithi079 ceph-mon[23699]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:32:14.942 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:32:14 smithi079 ceph-mon[30023]: from='client.15051 -' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:32:14.942 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:32:14 smithi079 ceph-mon[30023]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:32:15.176 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:32:14 smithi104 ceph-mon[25639]: from='client.15051 -' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:32:15.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:32:14 smithi104 ceph-mon[25639]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:32:16.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:32:16 smithi079 ceph-mon[23699]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:32:16.942 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:32:16 smithi079 ceph-mon[30023]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:32:17.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:32:16 smithi104 ceph-mon[25639]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:32:18.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:32:18 smithi104 ceph-mon[25639]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:32:18.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:32:18 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:32:18.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:32:18 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:32:18.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:32:18 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:32:18.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:32:18 smithi079 ceph-mon[23699]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:32:18.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:32:18 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:32:18.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:32:18 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:32:18.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:32:18 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:32:18.692 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:32:18 smithi079 ceph-mon[30023]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:32:18.692 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:32:18 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:32:18.692 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:32:18 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:32:18.693 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:32:18 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:32:20.676 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:32:20 smithi104 ceph-mon[25639]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:32:20.691 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:32:20 smithi079 ceph-mon[23699]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:32:20.692 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:32:20 smithi079 ceph-mon[30023]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:32:22.676 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:32:22 smithi104 ceph-mon[25639]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:32:22.691 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:32:22 smithi079 ceph-mon[23699]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:32:22.692 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:32:22 smithi079 ceph-mon[30023]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:32:23.427 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:32:23 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: ::ffff:172.21.15.104 - - [21/Sep/2024:12:32:23] "GET /metrics HTTP/1.1" 200 38510 "" "Prometheus/2.43.0" 2024-09-21T12:32:24.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:32:24 smithi104 ceph-mon[25639]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:32:24.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:32:24 smithi104 ceph-mon[25639]: from='client.15051 -' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:32:24.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:32:24 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T12:32:24.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:32:24 smithi079 ceph-mon[23699]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:32:24.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:32:24 smithi079 ceph-mon[23699]: from='client.15051 -' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:32:24.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:32:24 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T12:32:24.692 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:32:24 smithi079 ceph-mon[30023]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:32:24.692 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:32:24 smithi079 ceph-mon[30023]: from='client.15051 -' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:32:24.692 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:32:24 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T12:32:26.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:32:26 smithi104 ceph-mon[25639]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:32:26.691 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:32:26 smithi079 ceph-mon[23699]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:32:26.692 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:32:26 smithi079 ceph-mon[30023]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:32:28.666 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:32:28 smithi104 ceph-mon[25639]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:32:28.666 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:32:28 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:32:28.666 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:32:28 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:32:28.666 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:32:28 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:32:28.691 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:32:28 smithi079 ceph-mon[23699]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:32:28.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:32:28 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:32:28.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:32:28 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:32:28.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:32:28 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:32:28.692 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:32:28 smithi079 ceph-mon[30023]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:32:28.692 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:32:28 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:32:28.692 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:32:28 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:32:28.692 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:32:28 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:32:30.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:32:30 smithi104 ceph-mon[25639]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:32:30.691 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:32:30 smithi079 ceph-mon[23699]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:32:30.692 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:32:30 smithi079 ceph-mon[30023]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:32:32.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:32:32 smithi104 ceph-mon[25639]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:32:32.691 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:32:32 smithi079 ceph-mon[23699]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:32:32.692 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:32:32 smithi079 ceph-mon[30023]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:32:33.442 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:32:33 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: ::ffff:172.21.15.104 - - [21/Sep/2024:12:32:33] "GET /metrics HTTP/1.1" 200 38510 "" "Prometheus/2.43.0" 2024-09-21T12:32:34.676 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:32:34 smithi104 ceph-mon[25639]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:32:34.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:32:34 smithi104 ceph-mon[25639]: from='client.15051 -' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:32:34.691 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:32:34 smithi079 ceph-mon[23699]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:32:34.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:32:34 smithi079 ceph-mon[23699]: from='client.15051 -' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:32:34.692 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:32:34 smithi079 ceph-mon[30023]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:32:34.692 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:32:34 smithi079 ceph-mon[30023]: from='client.15051 -' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:32:36.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:32:36 smithi104 ceph-mon[25639]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:32:36.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:32:36 smithi079 ceph-mon[23699]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:32:36.692 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:32:36 smithi079 ceph-mon[30023]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:32:38.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:32:38 smithi104 ceph-mon[25639]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:32:38.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:32:38 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:32:38.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:32:38 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:32:38.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:32:38 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:32:38.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:32:38 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T12:32:38.941 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:32:38 smithi079 ceph-mon[23699]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:32:38.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:32:38 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:32:38.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:32:38 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:32:38.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:32:38 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:32:38.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:32:38 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T12:32:38.942 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:32:38 smithi079 ceph-mon[30023]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:32:38.942 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:32:38 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:32:38.942 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:32:38 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:32:38.942 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:32:38 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:32:38.942 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:32:38 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T12:32:40.676 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:32:40 smithi104 ceph-mon[25639]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:32:40.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:32:40 smithi079 ceph-mon[23699]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:32:40.942 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:32:40 smithi079 ceph-mon[30023]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:32:42.744 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:32:42 smithi079 ceph-mon[23699]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:32:42.744 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:32:42 smithi079 ceph-mon[30023]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:32:42.926 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:32:42 smithi104 ceph-mon[25639]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:32:43.442 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:32:43 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: ::ffff:172.21.15.104 - - [21/Sep/2024:12:32:43] "GET /metrics HTTP/1.1" 200 38511 "" "Prometheus/2.43.0" 2024-09-21T12:32:44.926 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:32:44 smithi104 ceph-mon[25639]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:32:44.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:32:44 smithi104 ceph-mon[25639]: from='client.15051 -' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:32:44.941 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:32:44 smithi079 ceph-mon[23699]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:32:44.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:32:44 smithi079 ceph-mon[23699]: from='client.15051 -' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:32:44.942 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:32:44 smithi079 ceph-mon[30023]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:32:44.942 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:32:44 smithi079 ceph-mon[30023]: from='client.15051 -' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:32:46.472 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 33168022-7814-11ef-baf6-efdc52797490 -e sha1=441784946648ff53dd310293f9dd752afeeea72a -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force' 2024-09-21T12:32:46.681 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:32:46.795 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:32:46 smithi079 ceph-mon[23699]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:32:46.795 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:32:46 smithi079 ceph-mon[30023]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:32:46.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:32:46 smithi104 ceph-mon[25639]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:32:48.850 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:32:48 smithi079 ceph-mon[23699]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:32:48.850 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:32:48 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:32:48.850 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:32:48 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:32:48.850 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:32:48 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:32:48.850 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:32:48 smithi079 ceph-mon[30023]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:32:48.850 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:32:48 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:32:48.850 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:32:48 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:32:48.850 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:32:48 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:32:48.873 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 33168022-7814-11ef-baf6-efdc52797490 -e sha1=441784946648ff53dd310293f9dd752afeeea72a -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force' 2024-09-21T12:32:48.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:32:48 smithi104 ceph-mon[25639]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:32:48.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:32:48 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:32:48.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:32:48 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:32:48.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:32:48 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:32:49.072 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:32:50.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:32:50 smithi079 ceph-mon[23699]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:32:50.692 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:32:50 smithi079 ceph-mon[30023]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:32:50.702 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:32:50 smithi104 ceph-mon[25639]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:32:51.377 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 33168022-7814-11ef-baf6-efdc52797490 -e sha1=441784946648ff53dd310293f9dd752afeeea72a -- bash -c 'ceph config set global log_to_journald false --force' 2024-09-21T12:32:51.575 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:32:52.662 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:32:52 smithi079 ceph-mon[23699]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:32:52.663 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:32:52 smithi079 ceph-mon[30023]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:32:52.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:32:52 smithi104 ceph-mon[25639]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:32:53.192 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:32:53 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: ::ffff:172.21.15.104 - - [21/Sep/2024:12:32:53] "GET /metrics HTTP/1.1" 200 38511 "" "Prometheus/2.43.0" 2024-09-21T12:32:53.819 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:32:53 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T12:32:53.819 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:32:53 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T12:32:53.840 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 33168022-7814-11ef-baf6-efdc52797490 -e sha1=441784946648ff53dd310293f9dd752afeeea72a -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1' 2024-09-21T12:32:53.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:32:53 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T12:32:54.039 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:32:54.692 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:32:54 smithi079 ceph-mon[30023]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:32:54.692 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:32:54 smithi079 ceph-mon[30023]: from='client.15051 -' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:32:54.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:32:54 smithi079 ceph-mon[23699]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:32:54.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:32:54 smithi079 ceph-mon[23699]: from='client.15051 -' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:32:54.926 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:32:54 smithi104 ceph-mon[25639]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:32:54.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:32:54 smithi104 ceph-mon[25639]: from='client.15051 -' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:32:55.753 INFO:teuthology.orchestra.run.smithi079.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a 2024-09-21T12:32:56.415 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-09-21T12:32:56.424 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi079.front.sepia.ceph.com 2024-09-21T12:32:56.425 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 33168022-7814-11ef-baf6-efdc52797490 -e sha1=441784946648ff53dd310293f9dd752afeeea72a -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done' 2024-09-21T12:32:56.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:32:56 smithi079 ceph-mon[23699]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:32:56.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:32:56 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:32:56.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:32:56 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:32:56.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:32:56 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:32:56.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:32:56 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:32:56.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:32:56 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:32:56.693 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:32:56 smithi079 ceph-mon[30023]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:32:56.693 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:32:56 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:32:56.693 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:32:56 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:32:56.693 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:32:56 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:32:56.693 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:32:56 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:32:56.693 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:32:56 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:32:56.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:32:56 smithi104 ceph-mon[25639]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:32:56.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:32:56 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:32:56.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:32:56 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:32:56.928 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:32:56 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:32:56.928 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:32:56 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:32:56.928 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:32:56 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:32:57.092 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:32:57.779 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:32:57 smithi079 ceph-mon[30023]: from='client.15177 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:32:57.779 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:32:57 smithi079 ceph-mon[30023]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a 2024-09-21T12:32:57.779 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:32:57 smithi079 ceph-mon[30023]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a 2024-09-21T12:32:57.779 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:32:57 smithi079 ceph-mon[23699]: from='client.15177 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:32:57.779 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:32:57 smithi079 ceph-mon[23699]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a 2024-09-21T12:32:57.779 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:32:57 smithi079 ceph-mon[23699]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a 2024-09-21T12:32:57.926 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:32:57 smithi104 ceph-mon[25639]: from='client.15177 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:32:57.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:32:57 smithi104 ceph-mon[25639]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a 2024-09-21T12:32:57.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:32:57 smithi104 ceph-mon[25639]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a 2024-09-21T12:32:58.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:32:58 smithi079 ceph-mon[23699]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:32:58.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:32:58 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:32:58.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:32:58 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:32:58.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:32:58 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:32:58.692 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:32:58 smithi079 ceph-mon[30023]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:32:58.692 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:32:58 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:32:58.692 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:32:58 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:32:58.692 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:32:58 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:32:58.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:32:58 smithi104 ceph-mon[25639]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:32:58.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:32:58 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:32:58.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:32:58 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:32:58.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:32:58 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:33:00.435 INFO:teuthology.orchestra.run.smithi079.stdout:true 2024-09-21T12:33:00.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:33:00 smithi104 ceph-mon[25639]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:33:00.941 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:33:00 smithi079 ceph-mon[23699]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:33:00.942 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:33:00 smithi079 ceph-mon[30023]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:33:01.036 INFO:teuthology.orchestra.run.smithi079.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T12:33:01.037 INFO:teuthology.orchestra.run.smithi079.stdout:agent.smithi079 smithi079 running 3s ago 8m - - 2024-09-21T12:33:01.037 INFO:teuthology.orchestra.run.smithi079.stdout:agent.smithi104 smithi104 running 13s ago 8m - - 2024-09-21T12:33:01.037 INFO:teuthology.orchestra.run.smithi079.stdout:alertmanager.a smithi079 *:9093,9094 running (3m) 3s ago 4m 28.2M - 0.25.0 c8568f914cd2 1451b4f42e50 2024-09-21T12:33:01.037 INFO:teuthology.orchestra.run.smithi079.stdout:grafana.a smithi104 *:3000 running (3m) 13s ago 3m 86.6M - 9.4.7 954c08fa6188 c85dd02f6519 2024-09-21T12:33:01.037 INFO:teuthology.orchestra.run.smithi079.stdout:iscsi.foo.smithi079.vlhrek smithi079 *:5000 running (2m) 3s ago 2m 51.1M - 3.8 0adab276c97e ddb66382ae41 2024-09-21T12:33:01.037 INFO:teuthology.orchestra.run.smithi079.stdout:mgr.x smithi104 *:8443,8765 running (7m) 13s ago 7m 439M - 18.2.4-871-g6a1bdf85 0adab276c97e b0c77ff77579 2024-09-21T12:33:01.037 INFO:teuthology.orchestra.run.smithi079.stdout:mgr.y smithi079 *:9283,8765 running (9m) 3s ago 9m 503M - 18.2.4-871-g6a1bdf85 0adab276c97e e34c6cf8a73d 2024-09-21T12:33:01.037 INFO:teuthology.orchestra.run.smithi079.stdout:mon.a smithi079 running (9m) 3s ago 9m 53.5M 2048M 18.2.4-871-g6a1bdf85 0adab276c97e 794d66ac29c5 2024-09-21T12:33:01.037 INFO:teuthology.orchestra.run.smithi079.stdout:mon.b smithi104 running (8m) 13s ago 8m 45.0M 2048M 18.2.4-871-g6a1bdf85 0adab276c97e 460a6e8db654 2024-09-21T12:33:01.037 INFO:teuthology.orchestra.run.smithi079.stdout:mon.c smithi079 running (8m) 3s ago 8m 43.8M 2048M 18.2.4-871-g6a1bdf85 0adab276c97e 4b837a3d8ec3 2024-09-21T12:33:01.037 INFO:teuthology.orchestra.run.smithi079.stdout:node-exporter.a smithi079 *:9100 running (4m) 3s ago 4m 16.0M - 1.5.0 0da6a335fe13 f7bef67502dc 2024-09-21T12:33:01.037 INFO:teuthology.orchestra.run.smithi079.stdout:node-exporter.b smithi104 *:9100 running (3m) 13s ago 3m 15.7M - 1.5.0 0da6a335fe13 3701f535d078 2024-09-21T12:33:01.037 INFO:teuthology.orchestra.run.smithi079.stdout:osd.0 smithi079 running (7m) 3s ago 7m 58.1M 4096M 18.2.4-871-g6a1bdf85 0adab276c97e 46b0cbf47ea9 2024-09-21T12:33:01.037 INFO:teuthology.orchestra.run.smithi079.stdout:osd.1 smithi079 running (7m) 3s ago 7m 52.1M 4096M 18.2.4-871-g6a1bdf85 0adab276c97e 6f56f34ef26f 2024-09-21T12:33:01.038 INFO:teuthology.orchestra.run.smithi079.stdout:osd.2 smithi079 running (6m) 3s ago 6m 56.5M 4096M 18.2.4-871-g6a1bdf85 0adab276c97e 9a4767c65578 2024-09-21T12:33:01.038 INFO:teuthology.orchestra.run.smithi079.stdout:osd.3 smithi079 running (6m) 3s ago 6m 48.7M 4096M 18.2.4-871-g6a1bdf85 0adab276c97e 1e7af3834ad3 2024-09-21T12:33:01.038 INFO:teuthology.orchestra.run.smithi079.stdout:osd.4 smithi104 running (6m) 13s ago 6m 61.4M 4096M 18.2.4-871-g6a1bdf85 0adab276c97e 2b027321f961 2024-09-21T12:33:01.038 INFO:teuthology.orchestra.run.smithi079.stdout:osd.5 smithi104 running (5m) 13s ago 5m 57.9M 4096M 18.2.4-871-g6a1bdf85 0adab276c97e 5e738fa75e97 2024-09-21T12:33:01.038 INFO:teuthology.orchestra.run.smithi079.stdout:osd.6 smithi104 running (5m) 13s ago 5m 56.1M 4096M 18.2.4-871-g6a1bdf85 0adab276c97e fd2852c1818e 2024-09-21T12:33:01.038 INFO:teuthology.orchestra.run.smithi079.stdout:osd.7 smithi104 running (4m) 13s ago 4m 49.7M 4096M 18.2.4-871-g6a1bdf85 0adab276c97e d2b958c30fc4 2024-09-21T12:33:01.038 INFO:teuthology.orchestra.run.smithi079.stdout:prometheus.a smithi104 *:9095 running (3m) 13s ago 4m 53.7M - 2.43.0 a07b618ecd1d b307869246ee 2024-09-21T12:33:01.038 INFO:teuthology.orchestra.run.smithi079.stdout:rgw.foo.smithi079.dfrddk smithi079 *:8000 running (2m) 3s ago 2m 75.8M - 18.2.4-871-g6a1bdf85 0adab276c97e 16a684a83189 2024-09-21T12:33:01.038 INFO:teuthology.orchestra.run.smithi079.stdout:rgw.foo.smithi104.qonbxe smithi104 *:8000 running (2m) 13s ago 2m 75.9M - 18.2.4-871-g6a1bdf85 0adab276c97e c9f39a22a235 2024-09-21T12:33:01.038 INFO:teuthology.orchestra.run.smithi079.stdout:rgw.smpl.smithi079.tydkfh smithi079 *:80 running (2m) 3s ago 2m 75.9M - 18.2.4-871-g6a1bdf85 0adab276c97e f59806e2a744 2024-09-21T12:33:01.038 INFO:teuthology.orchestra.run.smithi079.stdout:rgw.smpl.smithi104.riklnj smithi104 *:80 running (2m) 13s ago 2m 75.6M - 18.2.4-871-g6a1bdf85 0adab276c97e 47c316d12d6a 2024-09-21T12:33:01.392 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-21T12:33:01.392 INFO:teuthology.orchestra.run.smithi079.stdout: "mon": { 2024-09-21T12:33:01.392 INFO:teuthology.orchestra.run.smithi079.stdout: "ceph version 18.2.4-871-g6a1bdf85 (6a1bdf854bb5803efd58957907442aa01f41a8fc) reef (stable)": 3 2024-09-21T12:33:01.392 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-21T12:33:01.392 INFO:teuthology.orchestra.run.smithi079.stdout: "mgr": { 2024-09-21T12:33:01.392 INFO:teuthology.orchestra.run.smithi079.stdout: "ceph version 18.2.4-871-g6a1bdf85 (6a1bdf854bb5803efd58957907442aa01f41a8fc) reef (stable)": 2 2024-09-21T12:33:01.392 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-21T12:33:01.392 INFO:teuthology.orchestra.run.smithi079.stdout: "osd": { 2024-09-21T12:33:01.392 INFO:teuthology.orchestra.run.smithi079.stdout: "ceph version 18.2.4-871-g6a1bdf85 (6a1bdf854bb5803efd58957907442aa01f41a8fc) reef (stable)": 8 2024-09-21T12:33:01.393 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-21T12:33:01.393 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw": { 2024-09-21T12:33:01.393 INFO:teuthology.orchestra.run.smithi079.stdout: "ceph version 18.2.4-871-g6a1bdf85 (6a1bdf854bb5803efd58957907442aa01f41a8fc) reef (stable)": 4 2024-09-21T12:33:01.393 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-21T12:33:01.393 INFO:teuthology.orchestra.run.smithi079.stdout: "overall": { 2024-09-21T12:33:01.393 INFO:teuthology.orchestra.run.smithi079.stdout: "ceph version 18.2.4-871-g6a1bdf85 (6a1bdf854bb5803efd58957907442aa01f41a8fc) reef (stable)": 17 2024-09-21T12:33:01.393 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-21T12:33:01.393 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-21T12:33:01.691 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:33:01 smithi079 ceph-mon[23699]: from='client.15183 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:33:01.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:33:01 smithi079 ceph-mon[23699]: from='client.25055 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:33:01.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:33:01 smithi079 ceph-mon[23699]: from='client.? 172.21.15.79:0/1283608597' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T12:33:01.692 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:33:01 smithi079 ceph-mon[30023]: from='client.15183 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:33:01.692 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:33:01 smithi079 ceph-mon[30023]: from='client.25055 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:33:01.692 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:33:01 smithi079 ceph-mon[30023]: from='client.? 172.21.15.79:0/1283608597' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T12:33:01.697 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-21T12:33:01.697 INFO:teuthology.orchestra.run.smithi079.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a", 2024-09-21T12:33:01.697 INFO:teuthology.orchestra.run.smithi079.stdout: "in_progress": true, 2024-09-21T12:33:01.697 INFO:teuthology.orchestra.run.smithi079.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T12:33:01.697 INFO:teuthology.orchestra.run.smithi079.stdout: "services_complete": [], 2024-09-21T12:33:01.697 INFO:teuthology.orchestra.run.smithi079.stdout: "progress": "", 2024-09-21T12:33:01.697 INFO:teuthology.orchestra.run.smithi079.stdout: "message": "Doing first pull of quay.ceph.io/ceph-ci/ceph:441784946648ff53dd310293f9dd752afeeea72a image", 2024-09-21T12:33:01.697 INFO:teuthology.orchestra.run.smithi079.stdout: "is_paused": false 2024-09-21T12:33:01.698 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-21T12:33:01.896 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:33:01 smithi104 ceph-mon[25639]: from='client.15183 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:33:01.896 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:33:01 smithi104 ceph-mon[25639]: from='client.25055 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:33:01.897 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:33:01 smithi104 ceph-mon[25639]: from='client.? 172.21.15.79:0/1283608597' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T12:33:02.055 INFO:teuthology.orchestra.run.smithi079.stdout:HEALTH_OK 2024-09-21T12:33:02.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:33:02 smithi104 ceph-mon[25639]: from='client.15195 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:33:02.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:33:02 smithi104 ceph-mon[25639]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:33:02.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:33:02 smithi104 ceph-mon[25639]: from='client.25073 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:33:02.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:33:02 smithi104 ceph-mon[25639]: from='client.? 172.21.15.79:0/4081685691' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-21T12:33:02.941 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:33:02 smithi079 ceph-mon[23699]: from='client.15195 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:33:02.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:33:02 smithi079 ceph-mon[23699]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:33:02.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:33:02 smithi079 ceph-mon[23699]: from='client.25073 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:33:02.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:33:02 smithi079 ceph-mon[23699]: from='client.? 172.21.15.79:0/4081685691' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-21T12:33:02.942 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:33:02 smithi079 ceph-mon[30023]: from='client.15195 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:33:02.942 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:33:02 smithi079 ceph-mon[30023]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:33:02.942 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:33:02 smithi079 ceph-mon[30023]: from='client.25073 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:33:02.942 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:33:02 smithi079 ceph-mon[30023]: from='client.? 172.21.15.79:0/4081685691' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-21T12:33:03.442 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:33:03 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: ::ffff:172.21.15.104 - - [21/Sep/2024:12:33:03] "GET /metrics HTTP/1.1" 200 38509 "" "Prometheus/2.43.0" 2024-09-21T12:33:04.926 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:33:04 smithi104 ceph-mon[25639]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:33:04.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:33:04 smithi104 ceph-mon[25639]: from='client.15051 -' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:33:04.941 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:33:04 smithi079 ceph-mon[23699]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:33:04.941 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:33:04 smithi079 ceph-mon[23699]: from='client.15051 -' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:33:04.942 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:33:04 smithi079 ceph-mon[30023]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:33:04.942 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:33:04 smithi079 ceph-mon[30023]: from='client.15051 -' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:33:06.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:33:06 smithi104 ceph-mon[25639]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:33:06.941 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:33:06 smithi079 ceph-mon[23699]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:33:06.942 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:33:06 smithi079 ceph-mon[30023]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:33:08.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:33:08 smithi104 ceph-mon[25639]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:33:08.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:33:08 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:33:08.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:33:08 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:33:08.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:33:08 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:33:08.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:33:08 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T12:33:08.941 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:33:08 smithi079 ceph-mon[23699]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:33:08.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:33:08 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:33:08.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:33:08 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:33:08.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:33:08 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:33:08.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:33:08 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T12:33:08.942 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:33:08 smithi079 ceph-mon[30023]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:33:08.942 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:33:08 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:33:08.942 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:33:08 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:33:08.942 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:33:08 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:33:08.942 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:33:08 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T12:33:10.884 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:33:10 smithi104 ceph-mon[25639]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:33:10.941 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:33:10 smithi079 ceph-mon[23699]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:33:10.942 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:33:10 smithi079 ceph-mon[30023]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:33:12.894 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:33:12 smithi104 ceph-mon[25639]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:33:12.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:33:12 smithi079 ceph-mon[23699]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:33:12.942 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:33:12 smithi079 ceph-mon[30023]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:33:13.442 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:33:13 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: ::ffff:172.21.15.104 - - [21/Sep/2024:12:33:13] "GET /metrics HTTP/1.1" 200 38505 "" "Prometheus/2.43.0" 2024-09-21T12:33:14.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:33:14 smithi104 ceph-mon[25639]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:33:14.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:33:14 smithi104 ceph-mon[25639]: from='client.15051 -' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:33:14.941 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:33:14 smithi079 ceph-mon[23699]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:33:14.941 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:33:14 smithi079 ceph-mon[23699]: from='client.15051 -' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:33:14.942 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:33:14 smithi079 ceph-mon[30023]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:33:14.942 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:33:14 smithi079 ceph-mon[30023]: from='client.15051 -' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:33:16.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:33:16 smithi104 ceph-mon[25639]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:33:16.941 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:33:16 smithi079 ceph-mon[23699]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:33:16.942 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:33:16 smithi079 ceph-mon[30023]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:33:18.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:33:18 smithi104 ceph-mon[25639]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:33:18.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:33:18 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:33:18.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:33:18 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:33:18.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:33:18 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:33:18.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:33:18 smithi079 ceph-mon[23699]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:33:18.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:33:18 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:33:18.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:33:18 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:33:18.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:33:18 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:33:18.942 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:33:18 smithi079 ceph-mon[30023]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:33:18.942 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:33:18 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:33:18.942 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:33:18 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:33:18.942 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:33:18 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:33:20.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:33:20 smithi104 ceph-mon[25639]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:33:20.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:33:20 smithi079 ceph-mon[23699]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:33:20.942 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:33:20 smithi079 ceph-mon[30023]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:33:22.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:33:22 smithi104 ceph-mon[25639]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:33:22.941 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:33:22 smithi079 ceph-mon[23699]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:33:22.942 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:33:22 smithi079 ceph-mon[30023]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:33:23.442 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:33:23 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: ::ffff:172.21.15.104 - - [21/Sep/2024:12:33:23] "GET /metrics HTTP/1.1" 200 38505 "" "Prometheus/2.43.0" 2024-09-21T12:33:23.926 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:33:23 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T12:33:23.941 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:33:23 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T12:33:23.942 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:33:23 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T12:33:24.926 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:33:24 smithi104 ceph-mon[25639]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:33:24.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:33:24 smithi104 ceph-mon[25639]: from='client.15051 -' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:33:24.941 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:33:24 smithi079 ceph-mon[23699]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:33:24.941 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:33:24 smithi079 ceph-mon[23699]: from='client.15051 -' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:33:24.942 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:33:24 smithi079 ceph-mon[30023]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:33:24.942 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:33:24 smithi079 ceph-mon[30023]: from='client.15051 -' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:33:26.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:33:26 smithi104 ceph-mon[25639]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:33:26.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:33:26 smithi079 ceph-mon[23699]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:33:26.942 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:33:26 smithi079 ceph-mon[30023]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:33:28.676 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:33:28 smithi104 ceph-mon[25639]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:33:28.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:33:28 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:33:28.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:33:28 smithi104 ceph-mon[25639]: Upgrade: Target is version 19.3.0-5104-g44178494 (unknown) 2024-09-21T12:33:28.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:33:28 smithi104 ceph-mon[25639]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, digests ['quay.ceph.io/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5'] 2024-09-21T12:33:28.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:33:28 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:33:28.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:33:28 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:33:28.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:33:28 smithi104 ceph-mon[25639]: Upgrade: Need to upgrade myself (mgr.y) 2024-09-21T12:33:28.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:33:28 smithi104 ceph-mon[25639]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5 on smithi104 2024-09-21T12:33:28.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:33:28 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:33:28.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:33:28 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:33:28.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:33:28 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:33:28.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:33:28 smithi079 ceph-mon[23699]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:33:28.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:33:28 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:33:28.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:33:28 smithi079 ceph-mon[23699]: Upgrade: Target is version 19.3.0-5104-g44178494 (unknown) 2024-09-21T12:33:28.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:33:28 smithi079 ceph-mon[23699]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, digests ['quay.ceph.io/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5'] 2024-09-21T12:33:28.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:33:28 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:33:28.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:33:28 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:33:28.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:33:28 smithi079 ceph-mon[23699]: Upgrade: Need to upgrade myself (mgr.y) 2024-09-21T12:33:28.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:33:28 smithi079 ceph-mon[23699]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5 on smithi104 2024-09-21T12:33:28.693 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:33:28 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:33:28.693 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:33:28 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:33:28.693 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:33:28 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:33:28.693 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:33:28 smithi079 ceph-mon[30023]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:33:28.693 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:33:28 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:33:28.693 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:33:28 smithi079 ceph-mon[30023]: Upgrade: Target is version 19.3.0-5104-g44178494 (unknown) 2024-09-21T12:33:28.693 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:33:28 smithi079 ceph-mon[30023]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, digests ['quay.ceph.io/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5'] 2024-09-21T12:33:28.693 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:33:28 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:33:28.693 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:33:28 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:33:28.693 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:33:28 smithi079 ceph-mon[30023]: Upgrade: Need to upgrade myself (mgr.y) 2024-09-21T12:33:28.694 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:33:28 smithi079 ceph-mon[30023]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5 on smithi104 2024-09-21T12:33:28.694 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:33:28 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:33:28.694 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:33:28 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:33:28.694 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:33:28 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:33:30.562 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:33:30 smithi104 ceph-mon[25639]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:33:30.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:33:30 smithi079 ceph-mon[23699]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:33:30.692 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:33:30 smithi079 ceph-mon[30023]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:33:32.349 INFO:teuthology.orchestra.run.smithi079.stdout:true 2024-09-21T12:33:32.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:33:32 smithi104 ceph-mon[25639]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:33:32.691 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:33:32 smithi079 ceph-mon[23699]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:33:32.692 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:33:32 smithi079 ceph-mon[30023]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:33:32.877 INFO:teuthology.orchestra.run.smithi079.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T12:33:32.877 INFO:teuthology.orchestra.run.smithi079.stdout:agent.smithi079 smithi079 running 15s ago 9m - - 2024-09-21T12:33:32.877 INFO:teuthology.orchestra.run.smithi079.stdout:agent.smithi104 smithi104 running 4s ago 8m - - 2024-09-21T12:33:32.877 INFO:teuthology.orchestra.run.smithi079.stdout:alertmanager.a smithi079 *:9093,9094 running (3m) 15s ago 4m 28.2M - 0.25.0 c8568f914cd2 1451b4f42e50 2024-09-21T12:33:32.878 INFO:teuthology.orchestra.run.smithi079.stdout:grafana.a smithi104 *:3000 running (4m) 4s ago 4m 86.6M - 9.4.7 954c08fa6188 c85dd02f6519 2024-09-21T12:33:32.878 INFO:teuthology.orchestra.run.smithi079.stdout:iscsi.foo.smithi079.vlhrek smithi079 *:5000 running (2m) 15s ago 2m 51.1M - 3.8 0adab276c97e ddb66382ae41 2024-09-21T12:33:32.878 INFO:teuthology.orchestra.run.smithi079.stdout:mgr.x smithi104 *:8443,8765 running (8m) 4s ago 8m 440M - 18.2.4-871-g6a1bdf85 0adab276c97e b0c77ff77579 2024-09-21T12:33:32.878 INFO:teuthology.orchestra.run.smithi079.stdout:mgr.y smithi079 *:9283,8765 running (10m) 15s ago 10m 504M - 18.2.4-871-g6a1bdf85 0adab276c97e e34c6cf8a73d 2024-09-21T12:33:32.878 INFO:teuthology.orchestra.run.smithi079.stdout:mon.a smithi079 running (10m) 15s ago 10m 51.1M 2048M 18.2.4-871-g6a1bdf85 0adab276c97e 794d66ac29c5 2024-09-21T12:33:32.878 INFO:teuthology.orchestra.run.smithi079.stdout:mon.b smithi104 running (8m) 4s ago 8m 41.9M 2048M 18.2.4-871-g6a1bdf85 0adab276c97e 460a6e8db654 2024-09-21T12:33:32.878 INFO:teuthology.orchestra.run.smithi079.stdout:mon.c smithi079 running (8m) 15s ago 8m 40.2M 2048M 18.2.4-871-g6a1bdf85 0adab276c97e 4b837a3d8ec3 2024-09-21T12:33:32.878 INFO:teuthology.orchestra.run.smithi079.stdout:node-exporter.a smithi079 *:9100 running (4m) 15s ago 4m 15.8M - 1.5.0 0da6a335fe13 f7bef67502dc 2024-09-21T12:33:32.878 INFO:teuthology.orchestra.run.smithi079.stdout:node-exporter.b smithi104 *:9100 running (4m) 4s ago 4m 15.4M - 1.5.0 0da6a335fe13 3701f535d078 2024-09-21T12:33:32.878 INFO:teuthology.orchestra.run.smithi079.stdout:osd.0 smithi079 running (8m) 15s ago 8m 58.2M 4096M 18.2.4-871-g6a1bdf85 0adab276c97e 46b0cbf47ea9 2024-09-21T12:33:32.878 INFO:teuthology.orchestra.run.smithi079.stdout:osd.1 smithi079 running (7m) 15s ago 7m 52.3M 4096M 18.2.4-871-g6a1bdf85 0adab276c97e 6f56f34ef26f 2024-09-21T12:33:32.878 INFO:teuthology.orchestra.run.smithi079.stdout:osd.2 smithi079 running (7m) 15s ago 7m 56.6M 4096M 18.2.4-871-g6a1bdf85 0adab276c97e 9a4767c65578 2024-09-21T12:33:32.878 INFO:teuthology.orchestra.run.smithi079.stdout:osd.3 smithi079 running (6m) 15s ago 6m 49.0M 4096M 18.2.4-871-g6a1bdf85 0adab276c97e 1e7af3834ad3 2024-09-21T12:33:32.879 INFO:teuthology.orchestra.run.smithi079.stdout:osd.4 smithi104 running (6m) 4s ago 6m 61.7M 4096M 18.2.4-871-g6a1bdf85 0adab276c97e 2b027321f961 2024-09-21T12:33:32.879 INFO:teuthology.orchestra.run.smithi079.stdout:osd.5 smithi104 running (6m) 4s ago 6m 58.2M 4096M 18.2.4-871-g6a1bdf85 0adab276c97e 5e738fa75e97 2024-09-21T12:33:32.879 INFO:teuthology.orchestra.run.smithi079.stdout:osd.6 smithi104 running (5m) 4s ago 5m 56.3M 4096M 18.2.4-871-g6a1bdf85 0adab276c97e fd2852c1818e 2024-09-21T12:33:32.879 INFO:teuthology.orchestra.run.smithi079.stdout:osd.7 smithi104 running (5m) 4s ago 5m 50.0M 4096M 18.2.4-871-g6a1bdf85 0adab276c97e d2b958c30fc4 2024-09-21T12:33:32.879 INFO:teuthology.orchestra.run.smithi079.stdout:prometheus.a smithi104 *:9095 running (3m) 4s ago 5m 55.9M - 2.43.0 a07b618ecd1d b307869246ee 2024-09-21T12:33:32.879 INFO:teuthology.orchestra.run.smithi079.stdout:rgw.foo.smithi079.dfrddk smithi079 *:8000 running (3m) 15s ago 3m 76.0M - 18.2.4-871-g6a1bdf85 0adab276c97e 16a684a83189 2024-09-21T12:33:32.879 INFO:teuthology.orchestra.run.smithi079.stdout:rgw.foo.smithi104.qonbxe smithi104 *:8000 running (3m) 4s ago 3m 76.2M - 18.2.4-871-g6a1bdf85 0adab276c97e c9f39a22a235 2024-09-21T12:33:32.879 INFO:teuthology.orchestra.run.smithi079.stdout:rgw.smpl.smithi079.tydkfh smithi079 *:80 running (2m) 15s ago 2m 76.0M - 18.2.4-871-g6a1bdf85 0adab276c97e f59806e2a744 2024-09-21T12:33:32.879 INFO:teuthology.orchestra.run.smithi079.stdout:rgw.smpl.smithi104.riklnj smithi104 *:80 running (2m) 4s ago 2m 75.9M - 18.2.4-871-g6a1bdf85 0adab276c97e 47c316d12d6a 2024-09-21T12:33:33.191 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:33:33 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: ::ffff:172.21.15.104 - - [21/Sep/2024:12:33:33] "GET /metrics HTTP/1.1" 200 38506 "" "Prometheus/2.43.0" 2024-09-21T12:33:33.193 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-21T12:33:33.193 INFO:teuthology.orchestra.run.smithi079.stdout: "mon": { 2024-09-21T12:33:33.193 INFO:teuthology.orchestra.run.smithi079.stdout: "ceph version 18.2.4-871-g6a1bdf85 (6a1bdf854bb5803efd58957907442aa01f41a8fc) reef (stable)": 3 2024-09-21T12:33:33.193 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-21T12:33:33.193 INFO:teuthology.orchestra.run.smithi079.stdout: "mgr": { 2024-09-21T12:33:33.193 INFO:teuthology.orchestra.run.smithi079.stdout: "ceph version 18.2.4-871-g6a1bdf85 (6a1bdf854bb5803efd58957907442aa01f41a8fc) reef (stable)": 2 2024-09-21T12:33:33.193 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-21T12:33:33.193 INFO:teuthology.orchestra.run.smithi079.stdout: "osd": { 2024-09-21T12:33:33.193 INFO:teuthology.orchestra.run.smithi079.stdout: "ceph version 18.2.4-871-g6a1bdf85 (6a1bdf854bb5803efd58957907442aa01f41a8fc) reef (stable)": 8 2024-09-21T12:33:33.193 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-21T12:33:33.193 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw": { 2024-09-21T12:33:33.193 INFO:teuthology.orchestra.run.smithi079.stdout: "ceph version 18.2.4-871-g6a1bdf85 (6a1bdf854bb5803efd58957907442aa01f41a8fc) reef (stable)": 4 2024-09-21T12:33:33.193 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-21T12:33:33.193 INFO:teuthology.orchestra.run.smithi079.stdout: "overall": { 2024-09-21T12:33:33.194 INFO:teuthology.orchestra.run.smithi079.stdout: "ceph version 18.2.4-871-g6a1bdf85 (6a1bdf854bb5803efd58957907442aa01f41a8fc) reef (stable)": 17 2024-09-21T12:33:33.194 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-21T12:33:33.194 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-21T12:33:33.468 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-21T12:33:33.468 INFO:teuthology.orchestra.run.smithi079.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5", 2024-09-21T12:33:33.469 INFO:teuthology.orchestra.run.smithi079.stdout: "in_progress": true, 2024-09-21T12:33:33.469 INFO:teuthology.orchestra.run.smithi079.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T12:33:33.469 INFO:teuthology.orchestra.run.smithi079.stdout: "services_complete": [], 2024-09-21T12:33:33.469 INFO:teuthology.orchestra.run.smithi079.stdout: "progress": "0/23 daemons upgraded", 2024-09-21T12:33:33.469 INFO:teuthology.orchestra.run.smithi079.stdout: "message": "Pulling quay.ceph.io/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5 image on host smithi104", 2024-09-21T12:33:33.469 INFO:teuthology.orchestra.run.smithi079.stdout: "is_paused": false 2024-09-21T12:33:33.469 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-21T12:33:33.469 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:33:33 smithi079 ceph-mon[23699]: from='client.15219 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:33:33.469 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:33:33 smithi079 ceph-mon[23699]: from='client.15225 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:33:33.469 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:33:33 smithi079 ceph-mon[23699]: from='client.? 172.21.15.79:0/1147226619' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T12:33:33.469 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:33:33 smithi079 ceph-mon[30023]: from='client.15219 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:33:33.470 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:33:33 smithi079 ceph-mon[30023]: from='client.15225 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:33:33.470 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:33:33 smithi079 ceph-mon[30023]: from='client.? 172.21.15.79:0/1147226619' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T12:33:33.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:33:33 smithi104 ceph-mon[25639]: from='client.15219 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:33:33.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:33:33 smithi104 ceph-mon[25639]: from='client.15225 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:33:33.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:33:33 smithi104 ceph-mon[25639]: from='client.? 172.21.15.79:0/1147226619' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T12:33:33.821 INFO:teuthology.orchestra.run.smithi079.stdout:HEALTH_OK 2024-09-21T12:33:34.676 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:33:34 smithi104 ceph-mon[25639]: from='client.25097 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:33:34.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:33:34 smithi104 ceph-mon[25639]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:33:34.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:33:34 smithi104 ceph-mon[25639]: from='client.15051 -' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:33:34.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:33:34 smithi104 ceph-mon[25639]: from='client.25109 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:33:34.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:33:34 smithi104 ceph-mon[25639]: from='client.? 172.21.15.79:0/1414988088' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-21T12:33:34.692 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:33:34 smithi079 ceph-mon[30023]: from='client.25097 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:33:34.692 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:33:34 smithi079 ceph-mon[30023]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:33:34.692 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:33:34 smithi079 ceph-mon[30023]: from='client.15051 -' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:33:34.692 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:33:34 smithi079 ceph-mon[30023]: from='client.25109 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:33:34.692 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:33:34 smithi079 ceph-mon[30023]: from='client.? 172.21.15.79:0/1414988088' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-21T12:33:34.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:33:34 smithi079 ceph-mon[23699]: from='client.25097 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:33:34.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:33:34 smithi079 ceph-mon[23699]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:33:34.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:33:34 smithi079 ceph-mon[23699]: from='client.15051 -' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:33:34.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:33:34 smithi079 ceph-mon[23699]: from='client.25109 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:33:34.693 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:33:34 smithi079 ceph-mon[23699]: from='client.? 172.21.15.79:0/1414988088' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-21T12:33:36.676 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:33:36 smithi104 ceph-mon[25639]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:33:36.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:33:36 smithi079 ceph-mon[23699]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:33:36.692 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:33:36 smithi079 ceph-mon[30023]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:33:38.692 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:33:38 smithi079 ceph-mon[30023]: pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:33:38.692 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:33:38 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:33:38.692 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:33:38 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:33:38.692 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:33:38 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:33:38.692 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:33:38 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T12:33:38.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:33:38 smithi079 ceph-mon[23699]: pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:33:38.693 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:33:38 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:33:38.693 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:33:38 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:33:38.693 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:33:38 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:33:38.693 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:33:38 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T12:33:38.926 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:33:38 smithi104 ceph-mon[25639]: pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:33:38.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:33:38 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:33:38.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:33:38 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:33:38.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:33:38 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:33:38.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:33:38 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T12:33:40.692 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:33:40 smithi079 ceph-mon[30023]: pgmap v201: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:33:40.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:33:40 smithi079 ceph-mon[23699]: pgmap v201: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:33:40.926 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:33:40 smithi104 ceph-mon[25639]: pgmap v201: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:33:42.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:33:42 smithi079 ceph-mon[23699]: pgmap v202: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:33:42.692 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:33:42 smithi079 ceph-mon[30023]: pgmap v202: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:33:42.926 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:33:42 smithi104 ceph-mon[25639]: pgmap v202: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:33:43.441 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:33:43 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: ::ffff:172.21.15.104 - - [21/Sep/2024:12:33:43] "GET /metrics HTTP/1.1" 200 38509 "" "Prometheus/2.43.0" 2024-09-21T12:33:44.692 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:33:44 smithi079 ceph-mon[30023]: pgmap v203: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:33:44.692 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:33:44 smithi079 ceph-mon[30023]: from='client.15051 -' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:33:44.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:33:44 smithi079 ceph-mon[23699]: pgmap v203: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:33:44.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:33:44 smithi079 ceph-mon[23699]: from='client.15051 -' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:33:44.903 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:33:44 smithi104 ceph-mon[25639]: pgmap v203: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:33:44.903 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:33:44 smithi104 ceph-mon[25639]: from='client.15051 -' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:33:46.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:33:46 smithi079 ceph-mon[23699]: pgmap v204: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:33:46.692 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:33:46 smithi079 ceph-mon[30023]: pgmap v204: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:33:46.926 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:33:46 smithi104 ceph-mon[25639]: pgmap v204: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:33:48.676 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:33:48 smithi104 ceph-mon[25639]: pgmap v205: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:33:48.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:33:48 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:33:48.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:33:48 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:33:48.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:33:48 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:33:48.691 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:33:48 smithi079 ceph-mon[23699]: pgmap v205: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:33:48.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:33:48 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:33:48.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:33:48 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:33:48.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:33:48 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:33:48.692 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:33:48 smithi079 ceph-mon[30023]: pgmap v205: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:33:48.692 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:33:48 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:33:48.692 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:33:48 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:33:48.692 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:33:48 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:33:50.547 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:33:50 smithi079 ceph-mon[23699]: pgmap v206: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:33:50.547 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:33:50 smithi079 ceph-mon[30023]: pgmap v206: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:33:50.804 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:33:50 smithi104 ceph-mon[25639]: pgmap v206: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:33:52.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:33:52 smithi104 ceph-mon[25639]: pgmap v207: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:33:52.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:33:52 smithi079 ceph-mon[23699]: pgmap v207: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:33:52.942 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:33:52 smithi079 ceph-mon[30023]: pgmap v207: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:33:53.442 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:33:53 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: ::ffff:172.21.15.104 - - [21/Sep/2024:12:33:53] "GET /metrics HTTP/1.1" 200 38509 "" "Prometheus/2.43.0" 2024-09-21T12:33:53.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:33:53 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T12:33:53.941 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:33:53 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T12:33:53.942 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:33:53 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T12:33:54.789 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:33:54 smithi104 ceph-mon[25639]: pgmap v208: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:33:54.789 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:33:54 smithi104 ceph-mon[25639]: from='client.15051 -' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:33:54.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:33:54 smithi079 ceph-mon[23699]: pgmap v208: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:33:54.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:33:54 smithi079 ceph-mon[23699]: from='client.15051 -' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:33:54.942 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:33:54 smithi079 ceph-mon[30023]: pgmap v208: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:33:54.942 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:33:54 smithi079 ceph-mon[30023]: from='client.15051 -' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:33:56.574 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:33:56 smithi104 ceph-mon[25639]: pgmap v209: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:33:56.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:33:56 smithi079 ceph-mon[23699]: pgmap v209: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:33:56.942 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:33:56 smithi079 ceph-mon[30023]: pgmap v209: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:33:58.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:33:58 smithi104 ceph-mon[25639]: pgmap v210: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:33:58.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:33:58 smithi104 ceph-mon[25639]: Upgrade: Updating mgr.x 2024-09-21T12:33:58.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:33:58 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:33:58.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:33:58 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-21T12:33:58.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:33:58 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-21T12:33:58.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:33:58 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:33:58.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:33:58 smithi104 ceph-mon[25639]: Deploying daemon mgr.x on smithi104 2024-09-21T12:33:58.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:33:58 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:33:58.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:33:58 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:33:58.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:33:58 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:33:58.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:33:58 smithi079 ceph-mon[23699]: pgmap v210: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:33:58.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:33:58 smithi079 ceph-mon[23699]: Upgrade: Updating mgr.x 2024-09-21T12:33:58.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:33:58 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:33:58.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:33:58 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-21T12:33:58.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:33:58 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-21T12:33:58.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:33:58 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:33:58.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:33:58 smithi079 ceph-mon[23699]: Deploying daemon mgr.x on smithi104 2024-09-21T12:33:58.693 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:33:58 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:33:58.693 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:33:58 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:33:58.693 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:33:58 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:33:58.693 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:33:58 smithi079 ceph-mon[30023]: pgmap v210: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:33:58.693 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:33:58 smithi079 ceph-mon[30023]: Upgrade: Updating mgr.x 2024-09-21T12:33:58.693 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:33:58 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:33:58.693 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:33:58 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-21T12:33:58.693 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:33:58 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-21T12:33:58.693 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:33:58 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:33:58.693 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:33:58 smithi079 ceph-mon[30023]: Deploying daemon mgr.x on smithi104 2024-09-21T12:33:58.693 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:33:58 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:33:58.693 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:33:58 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:33:58.693 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:33:58 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:34:00.067 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:00 smithi104 systemd[1]: Stopping Ceph mgr.x for 33168022-7814-11ef-baf6-efdc52797490... 2024-09-21T12:34:00.666 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:00 smithi104 ceph-mon[25639]: pgmap v211: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:34:00.666 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:00 smithi104 podman[56191]: 2024-09-21 12:34:00.336144737 +0000 UTC m=+0.126988298 container died b0c77ff77579ecb197d757263ec61d6da3219696e0ce3597661f921fe41406ae (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, RELEASE=reef-6a1bdf8) 2024-09-21T12:34:00.666 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:00 smithi104 podman[56191]: 2024-09-21 12:34:00.664993412 +0000 UTC m=+0.455836965 container cleanup b0c77ff77579ecb197d757263ec61d6da3219696e0ce3597661f921fe41406ae (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, RELEASE=reef-6a1bdf8, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-09-21T12:34:00.691 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:00 smithi079 ceph-mon[23699]: pgmap v211: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:34:00.692 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:00 smithi079 ceph-mon[30023]: pgmap v211: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T12:34:00.926 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:00 smithi104 bash[56191]: ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x 2024-09-21T12:34:00.927 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:00 smithi104 podman[56204]: 2024-09-21 12:34:00.875065871 +0000 UTC m=+0.533577179 container remove b0c77ff77579ecb197d757263ec61d6da3219696e0ce3597661f921fe41406ae (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, RELEASE=reef-6a1bdf8, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T12:34:00.927 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:00 smithi104 systemd[1]: ceph-33168022-7814-11ef-baf6-efdc52797490@mgr.x.service: Main process exited, code=exited, status=143/n/a 2024-09-21T12:34:01.427 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:01 smithi104 systemd[1]: ceph-33168022-7814-11ef-baf6-efdc52797490@mgr.x.service: Failed with result 'exit-code'. 2024-09-21T12:34:01.427 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:01 smithi104 systemd[1]: Stopped Ceph mgr.x for 33168022-7814-11ef-baf6-efdc52797490. 2024-09-21T12:34:01.427 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:01 smithi104 systemd[1]: ceph-33168022-7814-11ef-baf6-efdc52797490@mgr.x.service: Consumed 11.057s CPU time. 2024-09-21T12:34:01.427 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:01 smithi104 systemd[1]: Starting Ceph mgr.x for 33168022-7814-11ef-baf6-efdc52797490... 2024-09-21T12:34:01.755 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:01 smithi104 podman[56333]: 2024-09-21 12:34:01.392145403 +0000 UTC m=+0.015657984 image pull b415816db8b41f544cdfdc71d193ff8b1ee008bf7b0ec4f64da8eb8f71b62e85 quay.ceph.io/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5 2024-09-21T12:34:01.755 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:01 smithi104 podman[56333]: 2024-09-21 12:34:01.517884416 +0000 UTC m=+0.141396995 container create 9038010f43b86a68e04abc5901b6b32af0e2e288e413ae8c9b8472a00239257b (image=quay.ceph.io/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-guits-main-2024-09-21-0842-4417849, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.build-date=20240731, GIT_CLEAN=True) 2024-09-21T12:34:02.177 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:01 smithi104 podman[56333]: 2024-09-21 12:34:01.75371308 +0000 UTC m=+0.377225658 container init 9038010f43b86a68e04abc5901b6b32af0e2e288e413ae8c9b8472a00239257b (image=quay.ceph.io/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-guits-main-2024-09-21-0842-4417849, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T12:34:02.177 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:01 smithi104 podman[56333]: 2024-09-21 12:34:01.756769663 +0000 UTC m=+0.380282235 container start 9038010f43b86a68e04abc5901b6b32af0e2e288e413ae8c9b8472a00239257b (image=quay.ceph.io/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-guits-main-2024-09-21-0842-4417849, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-09-21T12:34:02.177 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:01 smithi104 ceph-mgr[56352]: -- 172.21.15.104:0/2490900679 <== mon.1 v2:172.21.15.104:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55f14ec8d0e0 con 0x55f14ec6b000 2024-09-21T12:34:02.177 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:01 smithi104 bash[56333]: 9038010f43b86a68e04abc5901b6b32af0e2e288e413ae8c9b8472a00239257b 2024-09-21T12:34:02.177 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:01 smithi104 systemd[1]: Started Ceph mgr.x for 33168022-7814-11ef-baf6-efdc52797490. 2024-09-21T12:34:02.622 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:02 smithi079 ceph-mon[23699]: pgmap v212: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:34:02.622 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:02 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:34:02.622 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:02 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:34:02.622 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:02 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:34:02.622 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:02 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:34:02.622 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:02 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:34:02.622 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:02 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:34:02.622 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:02 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:34:02.623 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:02 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:34:02.623 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:02 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:34:02.623 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:02 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-09-21T12:34:02.623 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:02 smithi079 ceph-mon[23699]: Activating manager daemon x 2024-09-21T12:34:02.623 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:02 smithi079 ceph-mon[23699]: osdmap e95: 8 total, 8 up, 8 in 2024-09-21T12:34:02.623 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:02 smithi079 ceph-mon[23699]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd='[{"prefix": "mgr fail", "who": "y"}]': finished 2024-09-21T12:34:02.623 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:02 smithi079 ceph-mon[23699]: mgrmap e22: x(active, starting, since 0.00878085s) 2024-09-21T12:34:02.623 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:34:02 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: 2024-09-21T12:34:02.272+0000 7f040e516640 -1 mgr handle_mgr_map I was active but no longer am 2024-09-21T12:34:02.623 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:34:02 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: ignoring --setuser ceph since I am not root 2024-09-21T12:34:02.623 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:34:02 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: ignoring --setgroup ceph since I am not root 2024-09-21T12:34:02.623 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:34:02 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: 2024-09-21T12:34:02.420+0000 7f6fe84f0040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-21T12:34:02.624 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:34:02 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: 2024-09-21T12:34:02.475+0000 7f6fe84f0040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-21T12:34:02.624 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:02 smithi079 ceph-mon[30023]: pgmap v212: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:34:02.624 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:02 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:34:02.624 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:02 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:34:02.624 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:02 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:34:02.624 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:02 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:34:02.624 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:02 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:34:02.624 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:02 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:34:02.624 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:02 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:34:02.624 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:02 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:34:02.624 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:02 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:34:02.624 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:02 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-09-21T12:34:02.624 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:02 smithi079 ceph-mon[30023]: Activating manager daemon x 2024-09-21T12:34:02.625 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:02 smithi079 ceph-mon[30023]: osdmap e95: 8 total, 8 up, 8 in 2024-09-21T12:34:02.625 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:02 smithi079 ceph-mon[30023]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd='[{"prefix": "mgr fail", "who": "y"}]': finished 2024-09-21T12:34:02.625 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:02 smithi079 ceph-mon[30023]: mgrmap e22: x(active, starting, since 0.00878085s) 2024-09-21T12:34:02.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:02 smithi104 ceph-mon[25639]: pgmap v212: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:34:02.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:02 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:34:02.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:02 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:34:02.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:02 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:34:02.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:02 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:34:02.678 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:02 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:34:02.678 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:02 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:34:02.678 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:02 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:34:02.678 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:02 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' 2024-09-21T12:34:02.678 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:02 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:34:02.678 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:02 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-09-21T12:34:02.678 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:02 smithi104 ceph-mon[25639]: Activating manager daemon x 2024-09-21T12:34:02.678 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:02 smithi104 ceph-mon[25639]: osdmap e95: 8 total, 8 up, 8 in 2024-09-21T12:34:02.679 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:02 smithi104 ceph-mon[25639]: from='mgr.24341 172.21.15.79:0/1908902126' entity='mgr.y' cmd='[{"prefix": "mgr fail", "who": "y"}]': finished 2024-09-21T12:34:02.679 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:02 smithi104 ceph-mon[25639]: mgrmap e22: x(active, starting, since 0.00878085s) 2024-09-21T12:34:02.941 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:34:02 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: 2024-09-21T12:34:02.619+0000 7f6fe84f0040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-21T12:34:02.942 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:34:02 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: 2024-09-21T12:34:02.684+0000 7f6fe84f0040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-21T12:34:02.942 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:34:02 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: 2024-09-21T12:34:02.828+0000 7f6fe84f0040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-21T12:34:03.271 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:34:02 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: 2024-09-21T12:34:02.966+0000 7f6fe84f0040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-21T12:34:03.271 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:34:03 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: 2024-09-21T12:34:03.269+0000 7f6fe84f0040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-21T12:34:03.692 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:34:03 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: 2024-09-21T12:34:03.329+0000 7f6fe84f0040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-21T12:34:03.692 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:34:03 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: 2024-09-21T12:34:03.512+0000 7f6fe84f0040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-21T12:34:03.692 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Sep 21 12:34:03 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-alertmanager-a[54523]: ts=2024-09-21T12:34:03.507Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi079.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.79:8443: connect: connection refused" 2024-09-21T12:34:03.692 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Sep 21 12:34:03 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-alertmanager-a[54523]: ts=2024-09-21T12:34:03.507Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi104.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.104:8443: connect: connection refused" 2024-09-21T12:34:04.692 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:34:04 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: 2024-09-21T12:34:04.385+0000 7f6fe84f0040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-21T12:34:04.692 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:34:04 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: 2024-09-21T12:34:04.449+0000 7f6fe84f0040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-21T12:34:05.423 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:34:05 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: 2024-09-21T12:34:05.165+0000 7f6fe84f0040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-21T12:34:05.423 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:34:05 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: 2024-09-21T12:34:05.280+0000 7f6fe84f0040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-21T12:34:05.692 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:34:05 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: 2024-09-21T12:34:05.421+0000 7f6fe84f0040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-21T12:34:05.692 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:34:05 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: 2024-09-21T12:34:05.482+0000 7f6fe84f0040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-21T12:34:05.692 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:34:05 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: 2024-09-21T12:34:05.631+0000 7f6fe84f0040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-21T12:34:05.965 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:34:05 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: 2024-09-21T12:34:05.700+0000 7f6fe84f0040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-21T12:34:05.965 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:34:05 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: 2024-09-21T12:34:05.759+0000 7f6fe84f0040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-21T12:34:05.965 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:34:05 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: 2024-09-21T12:34:05.917+0000 7f6fe84f0040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-21T12:34:06.242 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:34:05 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: 2024-09-21T12:34:05.983+0000 7f6fe84f0040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-21T12:34:06.242 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:34:06 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: 2024-09-21T12:34:06.053+0000 7f6fe84f0040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-21T12:34:06.242 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:34:06 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: 2024-09-21T12:34:06.136+0000 7f6fe84f0040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-21T12:34:06.242 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:34:06 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: 2024-09-21T12:34:06.195+0000 7f6fe84f0040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-21T12:34:06.691 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:34:06 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: 2024-09-21T12:34:06.256+0000 7f6fe84f0040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-21T12:34:07.068 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:06 smithi079 ceph-mon[23699]: Active manager daemon x restarted 2024-09-21T12:34:07.068 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:06 smithi079 ceph-mon[23699]: Activating manager daemon x 2024-09-21T12:34:07.068 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:06 smithi079 ceph-mon[23699]: from='mgr.? 172.21.15.104:0/1289373660' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-21T12:34:07.068 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:06 smithi079 ceph-mon[23699]: from='mgr.? 172.21.15.104:0/1289373660' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-21T12:34:07.068 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:06 smithi079 ceph-mon[23699]: osdmap e96: 8 total, 8 up, 8 in 2024-09-21T12:34:07.068 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:06 smithi079 ceph-mon[23699]: mgrmap e23: x(active, starting, since 0.0116041s) 2024-09-21T12:34:07.068 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:06 smithi079 ceph-mon[23699]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-21T12:34:07.068 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:06 smithi079 ceph-mon[23699]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T12:34:07.068 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:06 smithi079 ceph-mon[23699]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T12:34:07.068 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:06 smithi079 ceph-mon[23699]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T12:34:07.068 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:06 smithi079 ceph-mon[23699]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-21T12:34:07.068 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:06 smithi079 ceph-mon[23699]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T12:34:07.069 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:06 smithi079 ceph-mon[23699]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T12:34:07.069 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:06 smithi079 ceph-mon[23699]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T12:34:07.069 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:06 smithi079 ceph-mon[23699]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T12:34:07.069 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:06 smithi079 ceph-mon[23699]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T12:34:07.069 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:06 smithi079 ceph-mon[23699]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T12:34:07.069 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:06 smithi079 ceph-mon[23699]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T12:34:07.069 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:06 smithi079 ceph-mon[23699]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T12:34:07.069 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:06 smithi079 ceph-mon[23699]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-21T12:34:07.069 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:06 smithi079 ceph-mon[23699]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-21T12:34:07.069 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:06 smithi079 ceph-mon[23699]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-21T12:34:07.069 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:06 smithi079 ceph-mon[23699]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-21T12:34:07.069 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:06 smithi079 ceph-mon[23699]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-21T12:34:07.069 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:06 smithi079 ceph-mon[30023]: Active manager daemon x restarted 2024-09-21T12:34:07.070 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:06 smithi079 ceph-mon[30023]: Activating manager daemon x 2024-09-21T12:34:07.070 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:06 smithi079 ceph-mon[30023]: from='mgr.? 172.21.15.104:0/1289373660' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-21T12:34:07.070 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:06 smithi079 ceph-mon[30023]: from='mgr.? 172.21.15.104:0/1289373660' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-21T12:34:07.070 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:06 smithi079 ceph-mon[30023]: osdmap e96: 8 total, 8 up, 8 in 2024-09-21T12:34:07.070 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:06 smithi079 ceph-mon[30023]: mgrmap e23: x(active, starting, since 0.0116041s) 2024-09-21T12:34:07.070 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:06 smithi079 ceph-mon[30023]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-21T12:34:07.070 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:06 smithi079 ceph-mon[30023]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T12:34:07.070 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:06 smithi079 ceph-mon[30023]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T12:34:07.070 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:06 smithi079 ceph-mon[30023]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T12:34:07.070 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:06 smithi079 ceph-mon[30023]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-21T12:34:07.070 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:06 smithi079 ceph-mon[30023]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T12:34:07.070 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:06 smithi079 ceph-mon[30023]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T12:34:07.070 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:06 smithi079 ceph-mon[30023]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T12:34:07.071 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:06 smithi079 ceph-mon[30023]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T12:34:07.071 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:06 smithi079 ceph-mon[30023]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T12:34:07.071 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:06 smithi079 ceph-mon[30023]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T12:34:07.071 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:06 smithi079 ceph-mon[30023]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T12:34:07.071 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:06 smithi079 ceph-mon[30023]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T12:34:07.071 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:06 smithi079 ceph-mon[30023]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-21T12:34:07.071 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:06 smithi079 ceph-mon[30023]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-21T12:34:07.071 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:06 smithi079 ceph-mon[30023]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-21T12:34:07.071 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:06 smithi079 ceph-mon[30023]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-21T12:34:07.071 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:06 smithi079 ceph-mon[30023]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-21T12:34:07.177 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:06 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: [21/Sep/2024:12:34:06] ENGINE Bus STARTING 2024-09-21T12:34:07.177 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:06 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: CherryPy Checker: 2024-09-21T12:34:07.177 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:06 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: The Application mounted at '' has an empty config. 2024-09-21T12:34:07.177 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:06 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: 2024-09-21T12:34:07.177 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:06 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: [21/Sep/2024:12:34:06] ENGINE Serving on http://:::9283 2024-09-21T12:34:07.177 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:06 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: [21/Sep/2024:12:34:06] ENGINE Bus STARTED 2024-09-21T12:34:07.177 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:06 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: [21/Sep/2024:12:34:06] ENGINE Bus STOPPING 2024-09-21T12:34:07.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:06 smithi104 ceph-mon[25639]: Active manager daemon x restarted 2024-09-21T12:34:07.178 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:06 smithi104 ceph-mon[25639]: Activating manager daemon x 2024-09-21T12:34:07.178 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:06 smithi104 ceph-mon[25639]: from='mgr.? 172.21.15.104:0/1289373660' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-21T12:34:07.178 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:06 smithi104 ceph-mon[25639]: from='mgr.? 172.21.15.104:0/1289373660' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-21T12:34:07.178 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:06 smithi104 ceph-mon[25639]: osdmap e96: 8 total, 8 up, 8 in 2024-09-21T12:34:07.178 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:06 smithi104 ceph-mon[25639]: mgrmap e23: x(active, starting, since 0.0116041s) 2024-09-21T12:34:07.178 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:06 smithi104 ceph-mon[25639]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-21T12:34:07.178 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:06 smithi104 ceph-mon[25639]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T12:34:07.178 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:06 smithi104 ceph-mon[25639]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T12:34:07.178 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:06 smithi104 ceph-mon[25639]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T12:34:07.178 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:06 smithi104 ceph-mon[25639]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-21T12:34:07.178 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:06 smithi104 ceph-mon[25639]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T12:34:07.178 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:06 smithi104 ceph-mon[25639]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T12:34:07.178 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:06 smithi104 ceph-mon[25639]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T12:34:07.179 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:06 smithi104 ceph-mon[25639]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T12:34:07.179 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:06 smithi104 ceph-mon[25639]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T12:34:07.179 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:06 smithi104 ceph-mon[25639]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T12:34:07.179 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:06 smithi104 ceph-mon[25639]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T12:34:07.179 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:06 smithi104 ceph-mon[25639]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T12:34:07.179 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:06 smithi104 ceph-mon[25639]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-21T12:34:07.179 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:06 smithi104 ceph-mon[25639]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-21T12:34:07.179 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:06 smithi104 ceph-mon[25639]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-21T12:34:07.179 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:06 smithi104 ceph-mon[25639]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-21T12:34:07.179 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:06 smithi104 ceph-mon[25639]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-21T12:34:07.441 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Sep 21 12:34:07 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-alertmanager-a[54523]: ts=2024-09-21T12:34:07.213Z caller=notify.go:743 level=info component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify success" attempts=5 2024-09-21T12:34:07.442 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:34:07 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: 2024-09-21T12:34:07.065+0000 7f6fe84f0040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-21T12:34:07.442 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:34:07 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: [21/Sep/2024:12:34:07] ENGINE Bus STARTING 2024-09-21T12:34:07.442 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:34:07 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: CherryPy Checker: 2024-09-21T12:34:07.442 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:34:07 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: The Application mounted at '' has an empty config. 2024-09-21T12:34:07.442 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:34:07 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: 2024-09-21T12:34:07.442 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:34:07 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: [21/Sep/2024:12:34:07] ENGINE Serving on http://:::9283 2024-09-21T12:34:07.442 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:34:07 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: [21/Sep/2024:12:34:07] ENGINE Bus STARTED 2024-09-21T12:34:07.924 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:07 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: [21/Sep/2024:12:34:07] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-21T12:34:07.924 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:07 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: [21/Sep/2024:12:34:07] ENGINE Bus STOPPED 2024-09-21T12:34:07.951 INFO:teuthology.orchestra.run.smithi079.stderr:Error ENOTSUP: Warning: due to ceph-mgr restart, some PG states may not be up to date 2024-09-21T12:34:07.951 INFO:teuthology.orchestra.run.smithi079.stderr:Module 'orchestrator' is not enabled/loaded (required by command 'orch upgrade status'): use `ceph mgr module enable orchestrator` to enable it 2024-09-21T12:34:08.177 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:07 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: 2024-09-21T12:34:07.947+0000 7fb3b536f640 -1 mgr.server reply reply (95) Operation not supported Warning: due to ceph-mgr restart, some PG states may not be up to date 2024-09-21T12:34:08.177 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:07 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: Module 'orchestrator' is not enabled/loaded (required by command 'orch upgrade status'): use `ceph mgr module enable orchestrator` to enable it 2024-09-21T12:34:08.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:07 smithi104 ceph-mon[25639]: Standby manager daemon y started 2024-09-21T12:34:08.178 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:07 smithi104 ceph-mon[25639]: from='mgr.? 172.21.15.79:0/1887417154' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-21T12:34:08.178 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:07 smithi104 ceph-mon[25639]: from='mgr.? 172.21.15.79:0/1887417154' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-21T12:34:08.178 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:07 smithi104 ceph-mon[25639]: from='mgr.? 172.21.15.79:0/1887417154' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-09-21T12:34:08.178 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:07 smithi104 ceph-mon[25639]: from='mgr.? 172.21.15.79:0/1887417154' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-21T12:34:08.178 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:07 smithi104 ceph-mon[25639]: Manager daemon x is now available 2024-09-21T12:34:08.191 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:07 smithi079 ceph-mon[23699]: Standby manager daemon y started 2024-09-21T12:34:08.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:07 smithi079 ceph-mon[23699]: from='mgr.? 172.21.15.79:0/1887417154' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-21T12:34:08.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:07 smithi079 ceph-mon[23699]: from='mgr.? 172.21.15.79:0/1887417154' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-21T12:34:08.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:07 smithi079 ceph-mon[23699]: from='mgr.? 172.21.15.79:0/1887417154' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-09-21T12:34:08.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:07 smithi079 ceph-mon[23699]: from='mgr.? 172.21.15.79:0/1887417154' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-21T12:34:08.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:07 smithi079 ceph-mon[23699]: Manager daemon x is now available 2024-09-21T12:34:08.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:07 smithi079 ceph-mon[30023]: Standby manager daemon y started 2024-09-21T12:34:08.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:07 smithi079 ceph-mon[30023]: from='mgr.? 172.21.15.79:0/1887417154' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-21T12:34:08.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:07 smithi079 ceph-mon[30023]: from='mgr.? 172.21.15.79:0/1887417154' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-21T12:34:08.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:07 smithi079 ceph-mon[30023]: from='mgr.? 172.21.15.79:0/1887417154' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-09-21T12:34:08.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:07 smithi079 ceph-mon[30023]: from='mgr.? 172.21.15.79:0/1887417154' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-21T12:34:08.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:07 smithi079 ceph-mon[30023]: Manager daemon x is now available 2024-09-21T12:34:08.631 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 33168022-7814-11ef-baf6-efdc52797490 -e sha1=441784946648ff53dd310293f9dd752afeeea72a -- bash -c 'ceph orch ps' 2024-09-21T12:34:08.843 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:34:09.176 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:08 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: [21/Sep/2024:12:34:08] ENGINE Bus STARTING 2024-09-21T12:34:09.177 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:08 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: [21/Sep/2024:12:34:08] ENGINE Serving on http://:::9283 2024-09-21T12:34:09.177 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:08 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: [21/Sep/2024:12:34:08] ENGINE Bus STARTED 2024-09-21T12:34:09.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:08 smithi104 ceph-mon[25639]: mgrmap e24: x(active, since 1.07142s), standbys: y 2024-09-21T12:34:09.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:08 smithi104 ceph-mon[25639]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-21T12:34:09.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:08 smithi104 ceph-mon[25639]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' 2024-09-21T12:34:09.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:08 smithi104 ceph-mon[25639]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' 2024-09-21T12:34:09.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:08 smithi104 ceph-mon[25639]: Migrating certs/keys for iscsi.foo spec to cert store 2024-09-21T12:34:09.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:08 smithi104 ceph-mon[25639]: Migrating certs/keys for rgw.foo spec to cert store 2024-09-21T12:34:09.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:08 smithi104 ceph-mon[25639]: Migrating certs/keys for rgw.smpl spec to cert store 2024-09-21T12:34:09.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:08 smithi104 ceph-mon[25639]: Checking for cert/key for grafana.a 2024-09-21T12:34:09.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:08 smithi104 ceph-mon[25639]: Migrating grafana.a cert to cert store 2024-09-21T12:34:09.178 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:08 smithi104 ceph-mon[25639]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' 2024-09-21T12:34:09.178 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:08 smithi104 ceph-mon[25639]: Migrating grafana.a key to cert store 2024-09-21T12:34:09.178 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:08 smithi104 ceph-mon[25639]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' 2024-09-21T12:34:09.178 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:08 smithi104 ceph-mon[25639]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' 2024-09-21T12:34:09.178 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:08 smithi104 ceph-mon[25639]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' 2024-09-21T12:34:09.178 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:08 smithi104 ceph-mon[25639]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:34:09.178 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:08 smithi104 ceph-mon[25639]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi104", "caps": []}]: dispatch 2024-09-21T12:34:09.178 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:08 smithi104 ceph-mon[25639]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi079", "caps": []}]: dispatch 2024-09-21T12:34:09.178 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:08 smithi104 ceph-mon[25639]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T12:34:09.178 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:08 smithi104 ceph-mon[25639]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T12:34:09.178 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:08 smithi104 ceph-mon[25639]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-09-21T12:34:09.178 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:08 smithi104 ceph-mon[25639]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-09-21T12:34:09.193 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:08 smithi079 ceph-mon[23699]: mgrmap e24: x(active, since 1.07142s), standbys: y 2024-09-21T12:34:09.193 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:08 smithi079 ceph-mon[23699]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-21T12:34:09.193 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:08 smithi079 ceph-mon[23699]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' 2024-09-21T12:34:09.193 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:08 smithi079 ceph-mon[23699]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' 2024-09-21T12:34:09.193 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:08 smithi079 ceph-mon[23699]: Migrating certs/keys for iscsi.foo spec to cert store 2024-09-21T12:34:09.193 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:08 smithi079 ceph-mon[23699]: Migrating certs/keys for rgw.foo spec to cert store 2024-09-21T12:34:09.193 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:08 smithi079 ceph-mon[23699]: Migrating certs/keys for rgw.smpl spec to cert store 2024-09-21T12:34:09.193 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:08 smithi079 ceph-mon[23699]: Checking for cert/key for grafana.a 2024-09-21T12:34:09.193 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:08 smithi079 ceph-mon[23699]: Migrating grafana.a cert to cert store 2024-09-21T12:34:09.193 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:08 smithi079 ceph-mon[23699]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' 2024-09-21T12:34:09.193 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:08 smithi079 ceph-mon[23699]: Migrating grafana.a key to cert store 2024-09-21T12:34:09.193 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:08 smithi079 ceph-mon[23699]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' 2024-09-21T12:34:09.193 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:08 smithi079 ceph-mon[23699]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' 2024-09-21T12:34:09.193 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:08 smithi079 ceph-mon[23699]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' 2024-09-21T12:34:09.194 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:08 smithi079 ceph-mon[23699]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:34:09.194 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:08 smithi079 ceph-mon[23699]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi104", "caps": []}]: dispatch 2024-09-21T12:34:09.194 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:08 smithi079 ceph-mon[23699]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi079", "caps": []}]: dispatch 2024-09-21T12:34:09.194 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:08 smithi079 ceph-mon[23699]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T12:34:09.194 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:08 smithi079 ceph-mon[23699]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T12:34:09.194 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:08 smithi079 ceph-mon[23699]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-09-21T12:34:09.194 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:08 smithi079 ceph-mon[23699]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-09-21T12:34:09.194 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:08 smithi079 ceph-mon[30023]: mgrmap e24: x(active, since 1.07142s), standbys: y 2024-09-21T12:34:09.194 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:08 smithi079 ceph-mon[30023]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-21T12:34:09.194 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:08 smithi079 ceph-mon[30023]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' 2024-09-21T12:34:09.194 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:08 smithi079 ceph-mon[30023]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' 2024-09-21T12:34:09.194 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:08 smithi079 ceph-mon[30023]: Migrating certs/keys for iscsi.foo spec to cert store 2024-09-21T12:34:09.194 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:08 smithi079 ceph-mon[30023]: Migrating certs/keys for rgw.foo spec to cert store 2024-09-21T12:34:09.195 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:08 smithi079 ceph-mon[30023]: Migrating certs/keys for rgw.smpl spec to cert store 2024-09-21T12:34:09.195 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:08 smithi079 ceph-mon[30023]: Checking for cert/key for grafana.a 2024-09-21T12:34:09.195 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:08 smithi079 ceph-mon[30023]: Migrating grafana.a cert to cert store 2024-09-21T12:34:09.195 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:08 smithi079 ceph-mon[30023]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' 2024-09-21T12:34:09.195 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:08 smithi079 ceph-mon[30023]: Migrating grafana.a key to cert store 2024-09-21T12:34:09.195 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:08 smithi079 ceph-mon[30023]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' 2024-09-21T12:34:09.195 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:08 smithi079 ceph-mon[30023]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' 2024-09-21T12:34:09.195 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:08 smithi079 ceph-mon[30023]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' 2024-09-21T12:34:09.195 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:08 smithi079 ceph-mon[30023]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:34:09.195 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:08 smithi079 ceph-mon[30023]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi104", "caps": []}]: dispatch 2024-09-21T12:34:09.195 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:08 smithi079 ceph-mon[30023]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi079", "caps": []}]: dispatch 2024-09-21T12:34:09.195 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:08 smithi079 ceph-mon[30023]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T12:34:09.195 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:08 smithi079 ceph-mon[30023]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T12:34:09.195 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:08 smithi079 ceph-mon[30023]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-09-21T12:34:09.195 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:08 smithi079 ceph-mon[30023]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-09-21T12:34:09.927 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:09 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: 2024-09-21T12:34:09.553+0000 7fb379ebc640 -1 log_channel(cephadm) log [ERR] : [21/Sep/2024:12:34:09] ENGINE Error in HTTPServer.serve 2024-09-21T12:34:09.927 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:09 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: Traceback (most recent call last): 2024-09-21T12:34:09.927 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:09 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T12:34:09.927 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:09 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: self._connections.run(self.expiration_interval) 2024-09-21T12:34:09.927 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:09 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T12:34:09.927 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:09 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: self._run(expiration_interval) 2024-09-21T12:34:09.927 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:09 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T12:34:09.927 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:09 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T12:34:09.928 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:09 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T12:34:09.928 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:09 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T12:34:09.928 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:09 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T12:34:09.928 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:09 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: s = self.context.wrap_socket( 2024-09-21T12:34:09.928 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:09 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T12:34:09.928 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:09 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: return self.sslsocket_class._create( 2024-09-21T12:34:09.928 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:09 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T12:34:09.928 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:09 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: self.do_handshake() 2024-09-21T12:34:09.928 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:09 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T12:34:09.928 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:09 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: self._sslobj.do_handshake() 2024-09-21T12:34:09.928 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:09 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T12:34:09.928 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:09 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: 2024-09-21T12:34:10.597 INFO:teuthology.orchestra.run.smithi079.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T12:34:10.598 INFO:teuthology.orchestra.run.smithi079.stdout:agent.smithi079 smithi079 running 12s ago 9m - - 2024-09-21T12:34:10.598 INFO:teuthology.orchestra.run.smithi079.stdout:agent.smithi104 smithi104 running 22s ago 9m - - 2024-09-21T12:34:10.598 INFO:teuthology.orchestra.run.smithi079.stdout:alertmanager.a smithi079 *:9093,9094 running (4m) 12s ago 5m 26.8M - 0.25.0 c8568f914cd2 1451b4f42e50 2024-09-21T12:34:10.598 INFO:teuthology.orchestra.run.smithi079.stdout:grafana.a smithi104 *:3000 running (4m) 22s ago 4m 88.0M - 9.4.7 954c08fa6188 c85dd02f6519 2024-09-21T12:34:10.598 INFO:teuthology.orchestra.run.smithi079.stdout:iscsi.foo.smithi079.vlhrek smithi079 *:5000 running (3m) 12s ago 3m 51.4M - 3.8 0adab276c97e ddb66382ae41 2024-09-21T12:34:10.598 INFO:teuthology.orchestra.run.smithi079.stdout:mgr.x smithi104 *:8443,9283,8765 starting - - - - 2024-09-21T12:34:10.598 INFO:teuthology.orchestra.run.smithi079.stdout:mgr.y smithi079 *:9283,8765 running (10m) 12s ago 10m 506M - 18.2.4-871-g6a1bdf85 0adab276c97e e34c6cf8a73d 2024-09-21T12:34:10.598 INFO:teuthology.orchestra.run.smithi079.stdout:mon.a smithi079 running (10m) 12s ago 10m 52.6M 2048M 18.2.4-871-g6a1bdf85 0adab276c97e 794d66ac29c5 2024-09-21T12:34:10.598 INFO:teuthology.orchestra.run.smithi079.stdout:mon.b smithi104 running (9m) 22s ago 9m 42.1M 2048M 18.2.4-871-g6a1bdf85 0adab276c97e 460a6e8db654 2024-09-21T12:34:10.598 INFO:teuthology.orchestra.run.smithi079.stdout:mon.c smithi079 running (9m) 12s ago 9m 40.8M 2048M 18.2.4-871-g6a1bdf85 0adab276c97e 4b837a3d8ec3 2024-09-21T12:34:10.598 INFO:teuthology.orchestra.run.smithi079.stdout:node-exporter.a smithi079 *:9100 running (5m) 12s ago 5m 16.1M - 1.5.0 0da6a335fe13 f7bef67502dc 2024-09-21T12:34:10.598 INFO:teuthology.orchestra.run.smithi079.stdout:node-exporter.b smithi104 *:9100 running (5m) 22s ago 5m 15.7M - 1.5.0 0da6a335fe13 3701f535d078 2024-09-21T12:34:10.598 INFO:teuthology.orchestra.run.smithi079.stdout:osd.0 smithi079 running (8m) 12s ago 8m 58.5M 4096M 18.2.4-871-g6a1bdf85 0adab276c97e 46b0cbf47ea9 2024-09-21T12:34:10.599 INFO:teuthology.orchestra.run.smithi079.stdout:osd.1 smithi079 running (8m) 12s ago 8m 52.6M 4096M 18.2.4-871-g6a1bdf85 0adab276c97e 6f56f34ef26f 2024-09-21T12:34:10.599 INFO:teuthology.orchestra.run.smithi079.stdout:osd.2 smithi079 running (7m) 12s ago 7m 56.8M 4096M 18.2.4-871-g6a1bdf85 0adab276c97e 9a4767c65578 2024-09-21T12:34:10.599 INFO:teuthology.orchestra.run.smithi079.stdout:osd.3 smithi079 running (7m) 12s ago 7m 49.4M 4096M 18.2.4-871-g6a1bdf85 0adab276c97e 1e7af3834ad3 2024-09-21T12:34:10.599 INFO:teuthology.orchestra.run.smithi079.stdout:osd.4 smithi104 running (7m) 22s ago 7m 61.8M 4096M 18.2.4-871-g6a1bdf85 0adab276c97e 2b027321f961 2024-09-21T12:34:10.599 INFO:teuthology.orchestra.run.smithi079.stdout:osd.5 smithi104 running (6m) 22s ago 6m 58.4M 4096M 18.2.4-871-g6a1bdf85 0adab276c97e 5e738fa75e97 2024-09-21T12:34:10.599 INFO:teuthology.orchestra.run.smithi079.stdout:osd.6 smithi104 running (6m) 22s ago 6m 56.4M 4096M 18.2.4-871-g6a1bdf85 0adab276c97e fd2852c1818e 2024-09-21T12:34:10.599 INFO:teuthology.orchestra.run.smithi079.stdout:osd.7 smithi104 running (6m) 22s ago 6m 50.2M 4096M 18.2.4-871-g6a1bdf85 0adab276c97e d2b958c30fc4 2024-09-21T12:34:10.599 INFO:teuthology.orchestra.run.smithi079.stdout:prometheus.a smithi104 *:9095 running (4m) 22s ago 5m 56.0M - 2.43.0 a07b618ecd1d b307869246ee 2024-09-21T12:34:10.599 INFO:teuthology.orchestra.run.smithi079.stdout:rgw.foo.smithi079.dfrddk smithi079 *:8000 running (3m) 12s ago 3m 76.4M - 18.2.4-871-g6a1bdf85 0adab276c97e 16a684a83189 2024-09-21T12:34:10.599 INFO:teuthology.orchestra.run.smithi079.stdout:rgw.foo.smithi104.qonbxe smithi104 *:8000 running (3m) 22s ago 3m 76.4M - 18.2.4-871-g6a1bdf85 0adab276c97e c9f39a22a235 2024-09-21T12:34:10.599 INFO:teuthology.orchestra.run.smithi079.stdout:rgw.smpl.smithi079.tydkfh smithi079 *:80 running (3m) 12s ago 3m 76.4M - 18.2.4-871-g6a1bdf85 0adab276c97e f59806e2a744 2024-09-21T12:34:10.599 INFO:teuthology.orchestra.run.smithi079.stdout:rgw.smpl.smithi104.riklnj smithi104 *:80 running (3m) 22s ago 3m 76.1M - 18.2.4-871-g6a1bdf85 0adab276c97e 47c316d12d6a 2024-09-21T12:34:10.933 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:10 smithi079 ceph-mon[23699]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T12:34:10.934 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:10 smithi079 ceph-mon[23699]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:34:10.934 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:10 smithi079 ceph-mon[23699]: Reconfiguring daemon agent.smithi104 on smithi104 2024-09-21T12:34:10.934 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:10 smithi079 ceph-mon[23699]: [21/Sep/2024:12:34:09] ENGINE Bus STARTING 2024-09-21T12:34:10.934 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:10 smithi079 ceph-mon[23699]: [21/Sep/2024:12:34:09] ENGINE Serving on https://172.21.15.104:7150 2024-09-21T12:34:10.934 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:10 smithi079 ceph-mon[23699]: [21/Sep/2024:12:34:09] ENGINE Error in HTTPServer.serve 2024-09-21T12:34:10.934 INFO:journalctl@ceph.mon.a.smithi079.stdout: Traceback (most recent call last): 2024-09-21T12:34:10.934 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T12:34:10.934 INFO:journalctl@ceph.mon.a.smithi079.stdout: self._connections.run(self.expiration_interval) 2024-09-21T12:34:10.934 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T12:34:10.934 INFO:journalctl@ceph.mon.a.smithi079.stdout: self._run(expiration_interval) 2024-09-21T12:34:10.934 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T12:34:10.934 INFO:journalctl@ceph.mon.a.smithi079.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T12:34:10.934 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T12:34:10.935 INFO:journalctl@ceph.mon.a.smithi079.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T12:34:10.935 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T12:34:10.935 INFO:journalctl@ceph.mon.a.smithi079.stdout: s = self.context.wrap_socket( 2024-09-21T12:34:10.935 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T12:34:10.935 INFO:journalctl@ceph.mon.a.smithi079.stdout: return self.sslsocket_class._create( 2024-09-21T12:34:10.935 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T12:34:10.935 INFO:journalctl@ceph.mon.a.smithi079.stdout: self.do_handshake() 2024-09-21T12:34:10.935 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T12:34:10.935 INFO:journalctl@ceph.mon.a.smithi079.stdout: self._sslobj.do_handshake() 2024-09-21T12:34:10.935 INFO:journalctl@ceph.mon.a.smithi079.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T12:34:10.935 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:10 smithi079 ceph-mon[23699]: [21/Sep/2024:12:34:09] ENGINE Serving on http://172.21.15.104:8765 2024-09-21T12:34:10.935 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:10 smithi079 ceph-mon[23699]: [21/Sep/2024:12:34:09] ENGINE Bus STARTED 2024-09-21T12:34:10.935 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:10 smithi079 ceph-mon[23699]: Deploying cephadm binary to smithi104 2024-09-21T12:34:10.935 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:10 smithi079 ceph-mon[23699]: mgrmap e25: x(active, since 2s), standbys: y 2024-09-21T12:34:10.936 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:10 smithi079 ceph-mon[30023]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T12:34:10.936 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:10 smithi079 ceph-mon[30023]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:34:10.936 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:10 smithi079 ceph-mon[30023]: Reconfiguring daemon agent.smithi104 on smithi104 2024-09-21T12:34:10.936 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:10 smithi079 ceph-mon[30023]: [21/Sep/2024:12:34:09] ENGINE Bus STARTING 2024-09-21T12:34:10.936 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:10 smithi079 ceph-mon[30023]: [21/Sep/2024:12:34:09] ENGINE Serving on https://172.21.15.104:7150 2024-09-21T12:34:10.936 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:10 smithi079 ceph-mon[30023]: [21/Sep/2024:12:34:09] ENGINE Error in HTTPServer.serve 2024-09-21T12:34:10.936 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2024-09-21T12:34:10.936 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T12:34:10.936 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._connections.run(self.expiration_interval) 2024-09-21T12:34:10.936 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T12:34:10.936 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._run(expiration_interval) 2024-09-21T12:34:10.936 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T12:34:10.936 INFO:journalctl@ceph.mon.c.smithi079.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T12:34:10.936 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T12:34:10.936 INFO:journalctl@ceph.mon.c.smithi079.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T12:34:10.937 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T12:34:10.937 INFO:journalctl@ceph.mon.c.smithi079.stdout: s = self.context.wrap_socket( 2024-09-21T12:34:10.937 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T12:34:10.937 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.sslsocket_class._create( 2024-09-21T12:34:10.937 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T12:34:10.937 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.do_handshake() 2024-09-21T12:34:10.937 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T12:34:10.937 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._sslobj.do_handshake() 2024-09-21T12:34:10.937 INFO:journalctl@ceph.mon.c.smithi079.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T12:34:10.937 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:10 smithi079 ceph-mon[30023]: [21/Sep/2024:12:34:09] ENGINE Serving on http://172.21.15.104:8765 2024-09-21T12:34:10.937 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:10 smithi079 ceph-mon[30023]: [21/Sep/2024:12:34:09] ENGINE Bus STARTED 2024-09-21T12:34:10.937 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:10 smithi079 ceph-mon[30023]: Deploying cephadm binary to smithi104 2024-09-21T12:34:10.937 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:10 smithi079 ceph-mon[30023]: mgrmap e25: x(active, since 2s), standbys: y 2024-09-21T12:34:11.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:10 smithi104 ceph-mon[25639]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T12:34:11.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:10 smithi104 ceph-mon[25639]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:34:11.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:10 smithi104 ceph-mon[25639]: Reconfiguring daemon agent.smithi104 on smithi104 2024-09-21T12:34:11.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:10 smithi104 ceph-mon[25639]: [21/Sep/2024:12:34:09] ENGINE Bus STARTING 2024-09-21T12:34:11.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:10 smithi104 ceph-mon[25639]: [21/Sep/2024:12:34:09] ENGINE Serving on https://172.21.15.104:7150 2024-09-21T12:34:11.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:10 smithi104 ceph-mon[25639]: [21/Sep/2024:12:34:09] ENGINE Error in HTTPServer.serve 2024-09-21T12:34:11.177 INFO:journalctl@ceph.mon.b.smithi104.stdout: Traceback (most recent call last): 2024-09-21T12:34:11.177 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T12:34:11.177 INFO:journalctl@ceph.mon.b.smithi104.stdout: self._connections.run(self.expiration_interval) 2024-09-21T12:34:11.177 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T12:34:11.177 INFO:journalctl@ceph.mon.b.smithi104.stdout: self._run(expiration_interval) 2024-09-21T12:34:11.177 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T12:34:11.178 INFO:journalctl@ceph.mon.b.smithi104.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T12:34:11.178 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T12:34:11.178 INFO:journalctl@ceph.mon.b.smithi104.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T12:34:11.178 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T12:34:11.178 INFO:journalctl@ceph.mon.b.smithi104.stdout: s = self.context.wrap_socket( 2024-09-21T12:34:11.178 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T12:34:11.178 INFO:journalctl@ceph.mon.b.smithi104.stdout: return self.sslsocket_class._create( 2024-09-21T12:34:11.178 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T12:34:11.178 INFO:journalctl@ceph.mon.b.smithi104.stdout: self.do_handshake() 2024-09-21T12:34:11.178 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T12:34:11.178 INFO:journalctl@ceph.mon.b.smithi104.stdout: self._sslobj.do_handshake() 2024-09-21T12:34:11.178 INFO:journalctl@ceph.mon.b.smithi104.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T12:34:11.178 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:10 smithi104 ceph-mon[25639]: [21/Sep/2024:12:34:09] ENGINE Serving on http://172.21.15.104:8765 2024-09-21T12:34:11.178 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:10 smithi104 ceph-mon[25639]: [21/Sep/2024:12:34:09] ENGINE Bus STARTED 2024-09-21T12:34:11.179 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:10 smithi104 ceph-mon[25639]: Deploying cephadm binary to smithi104 2024-09-21T12:34:11.179 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:10 smithi104 ceph-mon[25639]: mgrmap e25: x(active, since 2s), standbys: y 2024-09-21T12:34:11.355 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 33168022-7814-11ef-baf6-efdc52797490 -e sha1=441784946648ff53dd310293f9dd752afeeea72a -- bash -c 'ceph versions' 2024-09-21T12:34:11.555 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:34:11.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:11 smithi079 ceph-mon[23699]: Reconfiguring daemon agent.smithi079 on smithi079 2024-09-21T12:34:11.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:11 smithi079 ceph-mon[23699]: from='client.25151 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:34:11.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:11 smithi079 ceph-mon[23699]: Deploying cephadm binary to smithi079 2024-09-21T12:34:11.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:11 smithi079 ceph-mon[23699]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' 2024-09-21T12:34:11.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:11 smithi079 ceph-mon[23699]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' 2024-09-21T12:34:11.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:11 smithi079 ceph-mon[23699]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' 2024-09-21T12:34:11.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:11 smithi079 ceph-mon[23699]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi104", "name": "osd_memory_target"}]: dispatch 2024-09-21T12:34:11.942 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:11 smithi079 ceph-mon[30023]: Reconfiguring daemon agent.smithi079 on smithi079 2024-09-21T12:34:11.943 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:11 smithi079 ceph-mon[30023]: from='client.25151 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:34:11.943 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:11 smithi079 ceph-mon[30023]: Deploying cephadm binary to smithi079 2024-09-21T12:34:11.943 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:11 smithi079 ceph-mon[30023]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' 2024-09-21T12:34:11.943 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:11 smithi079 ceph-mon[30023]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' 2024-09-21T12:34:11.943 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:11 smithi079 ceph-mon[30023]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' 2024-09-21T12:34:11.943 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:11 smithi079 ceph-mon[30023]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi104", "name": "osd_memory_target"}]: dispatch 2024-09-21T12:34:12.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:11 smithi104 ceph-mon[25639]: Reconfiguring daemon agent.smithi079 on smithi079 2024-09-21T12:34:12.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:11 smithi104 ceph-mon[25639]: from='client.25151 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:34:12.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:11 smithi104 ceph-mon[25639]: Deploying cephadm binary to smithi079 2024-09-21T12:34:12.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:11 smithi104 ceph-mon[25639]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' 2024-09-21T12:34:12.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:11 smithi104 ceph-mon[25639]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' 2024-09-21T12:34:12.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:11 smithi104 ceph-mon[25639]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' 2024-09-21T12:34:12.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:11 smithi104 ceph-mon[25639]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi104", "name": "osd_memory_target"}]: dispatch 2024-09-21T12:34:12.677 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Sep 21 12:34:12 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-alertmanager-a[54523]: ts=2024-09-21T12:34:12.296Z caller=notify.go:743 level=info component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify success" attempts=7 2024-09-21T12:34:12.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:12 smithi079 ceph-mon[23699]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:34:12.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:12 smithi079 ceph-mon[23699]: mgrmap e26: x(active, since 4s), standbys: y 2024-09-21T12:34:12.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:12 smithi079 ceph-mon[23699]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' 2024-09-21T12:34:12.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:12 smithi079 ceph-mon[23699]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' 2024-09-21T12:34:12.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:12 smithi079 ceph-mon[23699]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' 2024-09-21T12:34:12.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:12 smithi079 ceph-mon[23699]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi079", "name": "osd_memory_target"}]: dispatch 2024-09-21T12:34:12.943 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:12 smithi079 ceph-mon[23699]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:34:12.943 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:12 smithi079 ceph-mon[23699]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:34:12.943 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:12 smithi079 ceph-mon[23699]: Updating smithi079:/etc/ceph/ceph.conf 2024-09-21T12:34:12.943 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:12 smithi079 ceph-mon[23699]: Updating smithi104:/etc/ceph/ceph.conf 2024-09-21T12:34:12.943 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:12 smithi079 ceph-mon[23699]: Updating smithi079:/var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T12:34:12.943 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:12 smithi079 ceph-mon[23699]: Updating smithi104:/var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T12:34:12.943 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:12 smithi079 ceph-mon[23699]: Updating smithi079:/etc/ceph/ceph.client.admin.keyring 2024-09-21T12:34:12.943 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:12 smithi079 ceph-mon[23699]: Updating smithi079:/var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T12:34:12.943 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:12 smithi079 ceph-mon[23699]: Updating smithi104:/etc/ceph/ceph.client.admin.keyring 2024-09-21T12:34:12.943 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:12 smithi079 ceph-mon[23699]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' 2024-09-21T12:34:12.943 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:12 smithi079 ceph-mon[23699]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' 2024-09-21T12:34:12.943 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:12 smithi079 ceph-mon[23699]: Updating smithi104:/var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T12:34:12.943 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:12 smithi079 ceph-mon[30023]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:34:12.944 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:12 smithi079 ceph-mon[30023]: mgrmap e26: x(active, since 4s), standbys: y 2024-09-21T12:34:12.944 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:12 smithi079 ceph-mon[30023]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' 2024-09-21T12:34:12.944 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:12 smithi079 ceph-mon[30023]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' 2024-09-21T12:34:12.944 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:12 smithi079 ceph-mon[30023]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' 2024-09-21T12:34:12.944 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:12 smithi079 ceph-mon[30023]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi079", "name": "osd_memory_target"}]: dispatch 2024-09-21T12:34:12.944 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:12 smithi079 ceph-mon[30023]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:34:12.944 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:12 smithi079 ceph-mon[30023]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:34:12.944 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:12 smithi079 ceph-mon[30023]: Updating smithi079:/etc/ceph/ceph.conf 2024-09-21T12:34:12.944 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:12 smithi079 ceph-mon[30023]: Updating smithi104:/etc/ceph/ceph.conf 2024-09-21T12:34:12.944 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:12 smithi079 ceph-mon[30023]: Updating smithi079:/var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T12:34:12.944 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:12 smithi079 ceph-mon[30023]: Updating smithi104:/var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T12:34:12.944 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:12 smithi079 ceph-mon[30023]: Updating smithi079:/etc/ceph/ceph.client.admin.keyring 2024-09-21T12:34:12.944 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:12 smithi079 ceph-mon[30023]: Updating smithi079:/var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T12:34:12.944 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:12 smithi079 ceph-mon[30023]: Updating smithi104:/etc/ceph/ceph.client.admin.keyring 2024-09-21T12:34:12.944 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:12 smithi079 ceph-mon[30023]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' 2024-09-21T12:34:12.945 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:12 smithi079 ceph-mon[30023]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' 2024-09-21T12:34:12.945 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:12 smithi079 ceph-mon[30023]: Updating smithi104:/var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T12:34:13.176 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:12 smithi104 ceph-mon[25639]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:34:13.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:12 smithi104 ceph-mon[25639]: mgrmap e26: x(active, since 4s), standbys: y 2024-09-21T12:34:13.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:12 smithi104 ceph-mon[25639]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' 2024-09-21T12:34:13.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:12 smithi104 ceph-mon[25639]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' 2024-09-21T12:34:13.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:12 smithi104 ceph-mon[25639]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' 2024-09-21T12:34:13.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:12 smithi104 ceph-mon[25639]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi079", "name": "osd_memory_target"}]: dispatch 2024-09-21T12:34:13.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:12 smithi104 ceph-mon[25639]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:34:13.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:12 smithi104 ceph-mon[25639]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:34:13.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:12 smithi104 ceph-mon[25639]: Updating smithi079:/etc/ceph/ceph.conf 2024-09-21T12:34:13.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:12 smithi104 ceph-mon[25639]: Updating smithi104:/etc/ceph/ceph.conf 2024-09-21T12:34:13.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:12 smithi104 ceph-mon[25639]: Updating smithi079:/var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T12:34:13.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:12 smithi104 ceph-mon[25639]: Updating smithi104:/var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T12:34:13.178 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:12 smithi104 ceph-mon[25639]: Updating smithi079:/etc/ceph/ceph.client.admin.keyring 2024-09-21T12:34:13.178 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:12 smithi104 ceph-mon[25639]: Updating smithi079:/var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T12:34:13.178 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:12 smithi104 ceph-mon[25639]: Updating smithi104:/etc/ceph/ceph.client.admin.keyring 2024-09-21T12:34:13.178 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:12 smithi104 ceph-mon[25639]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' 2024-09-21T12:34:13.178 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:12 smithi104 ceph-mon[25639]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' 2024-09-21T12:34:13.178 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:12 smithi104 ceph-mon[25639]: Updating smithi104:/var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T12:34:13.275 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:34:13 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: ::ffff:172.21.15.104 - - [21/Sep/2024:12:34:13] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-21T12:34:13.486 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-21T12:34:13.486 INFO:teuthology.orchestra.run.smithi079.stdout: "mon": { 2024-09-21T12:34:13.486 INFO:teuthology.orchestra.run.smithi079.stdout: "ceph version 18.2.4-871-g6a1bdf85 (6a1bdf854bb5803efd58957907442aa01f41a8fc) reef (stable)": 3 2024-09-21T12:34:13.486 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-21T12:34:13.486 INFO:teuthology.orchestra.run.smithi079.stdout: "mgr": { 2024-09-21T12:34:13.486 INFO:teuthology.orchestra.run.smithi079.stdout: "ceph version 18.2.4-871-g6a1bdf85 (6a1bdf854bb5803efd58957907442aa01f41a8fc) reef (stable)": 1, 2024-09-21T12:34:13.486 INFO:teuthology.orchestra.run.smithi079.stdout: "ceph version 19.3.0-5104-g44178494 (441784946648ff53dd310293f9dd752afeeea72a) squid (dev)": 1 2024-09-21T12:34:13.486 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-21T12:34:13.487 INFO:teuthology.orchestra.run.smithi079.stdout: "osd": { 2024-09-21T12:34:13.487 INFO:teuthology.orchestra.run.smithi079.stdout: "ceph version 18.2.4-871-g6a1bdf85 (6a1bdf854bb5803efd58957907442aa01f41a8fc) reef (stable)": 8 2024-09-21T12:34:13.487 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-21T12:34:13.487 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw": { 2024-09-21T12:34:13.487 INFO:teuthology.orchestra.run.smithi079.stdout: "ceph version 18.2.4-871-g6a1bdf85 (6a1bdf854bb5803efd58957907442aa01f41a8fc) reef (stable)": 4 2024-09-21T12:34:13.487 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-21T12:34:13.487 INFO:teuthology.orchestra.run.smithi079.stdout: "overall": { 2024-09-21T12:34:13.487 INFO:teuthology.orchestra.run.smithi079.stdout: "ceph version 18.2.4-871-g6a1bdf85 (6a1bdf854bb5803efd58957907442aa01f41a8fc) reef (stable)": 16, 2024-09-21T12:34:13.487 INFO:teuthology.orchestra.run.smithi079.stdout: "ceph version 19.3.0-5104-g44178494 (441784946648ff53dd310293f9dd752afeeea72a) squid (dev)": 1 2024-09-21T12:34:13.487 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-21T12:34:13.487 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-21T12:34:14.168 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:13 smithi079 ceph-mon[23699]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' 2024-09-21T12:34:14.169 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:13 smithi079 ceph-mon[23699]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' 2024-09-21T12:34:14.169 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:13 smithi079 ceph-mon[23699]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' 2024-09-21T12:34:14.169 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:13 smithi079 ceph-mon[23699]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' 2024-09-21T12:34:14.169 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:13 smithi079 ceph-mon[23699]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:34:14.169 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:13 smithi079 ceph-mon[23699]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi079", "caps": []}]: dispatch 2024-09-21T12:34:14.169 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:13 smithi079 ceph-mon[23699]: from='client.? 172.21.15.79:0/2175334486' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T12:34:14.169 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:13 smithi079 ceph-mon[23699]: Deploying daemon agent.smithi079 on smithi079 2024-09-21T12:34:14.169 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:13 smithi079 ceph-mon[30023]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' 2024-09-21T12:34:14.169 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:13 smithi079 ceph-mon[30023]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' 2024-09-21T12:34:14.169 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:13 smithi079 ceph-mon[30023]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' 2024-09-21T12:34:14.170 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:13 smithi079 ceph-mon[30023]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' 2024-09-21T12:34:14.170 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:13 smithi079 ceph-mon[30023]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:34:14.170 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:13 smithi079 ceph-mon[30023]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi079", "caps": []}]: dispatch 2024-09-21T12:34:14.170 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:13 smithi079 ceph-mon[30023]: from='client.? 172.21.15.79:0/2175334486' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T12:34:14.170 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:13 smithi079 ceph-mon[30023]: Deploying daemon agent.smithi079 on smithi079 2024-09-21T12:34:14.176 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:13 smithi104 ceph-mon[25639]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' 2024-09-21T12:34:14.176 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:13 smithi104 ceph-mon[25639]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' 2024-09-21T12:34:14.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:13 smithi104 ceph-mon[25639]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' 2024-09-21T12:34:14.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:13 smithi104 ceph-mon[25639]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' 2024-09-21T12:34:14.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:13 smithi104 ceph-mon[25639]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:34:14.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:13 smithi104 ceph-mon[25639]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi079", "caps": []}]: dispatch 2024-09-21T12:34:14.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:13 smithi104 ceph-mon[25639]: from='client.? 172.21.15.79:0/2175334486' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T12:34:14.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:13 smithi104 ceph-mon[25639]: Deploying daemon agent.smithi079 on smithi079 2024-09-21T12:34:15.073 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 33168022-7814-11ef-baf6-efdc52797490 -e sha1=441784946648ff53dd310293f9dd752afeeea72a -- bash -c 'echo "wait for servicemap items w/ changing names to refresh"' 2024-09-21T12:34:15.453 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:34:16.151 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:15 smithi079 ceph-mon[23699]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-21T12:34:16.151 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:15 smithi079 ceph-mon[30023]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-21T12:34:16.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:15 smithi104 ceph-mon[25639]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-21T12:34:17.455 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:17 smithi079 ceph-mon[23699]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' 2024-09-21T12:34:17.456 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:17 smithi079 ceph-mon[23699]: Reconfiguring iscsi.foo.smithi079.vlhrek (dependencies changed)... 2024-09-21T12:34:17.456 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:17 smithi079 ceph-mon[23699]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi079.vlhrek", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-21T12:34:17.456 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:17 smithi079 ceph-mon[23699]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:34:17.456 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:17 smithi079 ceph-mon[23699]: Reconfiguring daemon iscsi.foo.smithi079.vlhrek on smithi079 2024-09-21T12:34:17.456 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:17 smithi079 ceph-mon[30023]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' 2024-09-21T12:34:17.456 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:17 smithi079 ceph-mon[30023]: Reconfiguring iscsi.foo.smithi079.vlhrek (dependencies changed)... 2024-09-21T12:34:17.456 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:17 smithi079 ceph-mon[30023]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi079.vlhrek", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-21T12:34:17.456 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:17 smithi079 ceph-mon[30023]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:34:17.456 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:17 smithi079 ceph-mon[30023]: Reconfiguring daemon iscsi.foo.smithi079.vlhrek on smithi079 2024-09-21T12:34:17.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:17 smithi104 ceph-mon[25639]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' 2024-09-21T12:34:17.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:17 smithi104 ceph-mon[25639]: Reconfiguring iscsi.foo.smithi079.vlhrek (dependencies changed)... 2024-09-21T12:34:17.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:17 smithi104 ceph-mon[25639]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi079.vlhrek", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-21T12:34:17.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:17 smithi104 ceph-mon[25639]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:34:17.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:17 smithi104 ceph-mon[25639]: Reconfiguring daemon iscsi.foo.smithi079.vlhrek on smithi079 2024-09-21T12:34:17.828 INFO:teuthology.orchestra.run.smithi079.stdout:wait for servicemap items w/ changing names to refresh 2024-09-21T12:34:18.926 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:18 smithi079 ceph-mon[23699]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-21T12:34:18.926 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:18 smithi079 ceph-mon[23699]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' 2024-09-21T12:34:18.926 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:18 smithi079 ceph-mon[30023]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-21T12:34:18.926 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:18 smithi079 ceph-mon[30023]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' 2024-09-21T12:34:18.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:18 smithi104 ceph-mon[25639]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-21T12:34:18.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:18 smithi104 ceph-mon[25639]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' 2024-09-21T12:34:19.485 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 33168022-7814-11ef-baf6-efdc52797490 -e sha1=441784946648ff53dd310293f9dd752afeeea72a -- bash -c 'sleep 60' 2024-09-21T12:34:19.864 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:19 smithi079 ceph-mon[23699]: from='client.15051 -' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:34:19.865 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:19 smithi079 ceph-mon[30023]: from='client.15051 -' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:34:19.926 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:19 smithi104 ceph-mon[25639]: from='client.15051 -' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:34:20.828 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:20 smithi104 ceph-mon[25639]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T12:34:20.845 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:20 smithi079 ceph-mon[23699]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T12:34:20.845 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:20 smithi079 ceph-mon[30023]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T12:34:22.158 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:34:22.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:22 smithi079 ceph-mon[23699]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 6 op/s 2024-09-21T12:34:22.692 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:22 smithi079 ceph-mon[30023]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 6 op/s 2024-09-21T12:34:22.842 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:22 smithi104 ceph-mon[25639]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 6 op/s 2024-09-21T12:34:23.192 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:34:23 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[23924]: ::ffff:172.21.15.104 - - [21/Sep/2024:12:34:23] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-21T12:34:24.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:24 smithi079 ceph-mon[23699]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 6 op/s 2024-09-21T12:34:24.692 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:24 smithi079 ceph-mon[23699]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T12:34:24.692 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:24 smithi079 ceph-mon[30023]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 6 op/s 2024-09-21T12:34:24.693 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:24 smithi079 ceph-mon[30023]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T12:34:24.777 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:24 smithi104 ceph-mon[25639]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 6 op/s 2024-09-21T12:34:24.777 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:24 smithi104 ceph-mon[25639]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T12:34:24.777 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:24 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: 2024-09-21T12:34:24.552+0000 7fb3ad11f640 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi079.vlhrek ... 2024-09-21T12:34:24.778 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:24 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: Creating ceph-iscsi config... 2024-09-21T12:34:24.778 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:24 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: Write file: /var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/iscsi.foo.smithi079.vlhrek/iscsi-gateway.cfg 2024-09-21T12:34:24.778 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:24 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: Write file: /var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/iscsi.foo.smithi079.vlhrek/tcmu-runner-entrypoint.sh 2024-09-21T12:34:24.778 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:24 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: Non-zero exit code 1 from systemctl restart ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek 2024-09-21T12:34:24.778 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:24 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: systemctl: stderr Job for ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service failed because the control process exited with error code. 2024-09-21T12:34:24.778 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:24 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: systemctl: stderr See "systemctl status ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service" and "journalctl -xeu ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service" for details. 2024-09-21T12:34:24.778 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:24 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: Traceback (most recent call last): 2024-09-21T12:34:24.778 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:24 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-21T12:34:24.778 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:24 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: return _run_code(code, main_globals, None, 2024-09-21T12:34:24.778 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:24 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-21T12:34:24.778 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:24 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: exec(code, run_globals) 2024-09-21T12:34:24.779 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:24 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 5628, in 2024-09-21T12:34:24.779 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:24 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 5616, in main 2024-09-21T12:34:24.779 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:24 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 3090, in command_deploy_from 2024-09-21T12:34:24.779 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:24 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 3125, in _common_deploy 2024-09-21T12:34:24.779 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:24 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 3145, in _deploy_daemon_container 2024-09-21T12:34:24.779 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:24 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 1147, in deploy_daemon 2024-09-21T12:34:24.779 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:24 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: File "/tmp/tmptpwpfqo2.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-21T12:34:24.779 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:24 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: RuntimeError: Failed command: systemctl restart ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek: Job for ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service failed because the control process exited with error code. 2024-09-21T12:34:24.779 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:24 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: See "systemctl status ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service" and "journalctl -xeu ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service" for details. 2024-09-21T12:34:24.780 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:24 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: Traceback (most recent call last): 2024-09-21T12:34:24.780 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:24 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1164, in _check_daemons 2024-09-21T12:34:24.780 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:24 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: self.mgr._daemon_action(daemon_spec, action=action) 2024-09-21T12:34:24.780 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:24 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2610, in _daemon_action 2024-09-21T12:34:24.780 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:24 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: return self.wait_async( 2024-09-21T12:34:24.780 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:24 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: File "/usr/share/ceph/mgr/cephadm/module.py", line 878, in wait_async 2024-09-21T12:34:24.780 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:24 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: return self.event_loop.get_result(coro, timeout) 2024-09-21T12:34:24.780 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:24 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-09-21T12:34:24.780 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:24 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: return future.result(timeout) 2024-09-21T12:34:24.780 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:24 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: File "/lib64/python3.9/concurrent/futures/_base.py", line 446, in result 2024-09-21T12:34:24.780 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:24 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: return self.__get_result() 2024-09-21T12:34:24.780 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:24 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: File "/lib64/python3.9/concurrent/futures/_base.py", line 391, in __get_result 2024-09-21T12:34:24.780 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:24 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: raise self._exception 2024-09-21T12:34:24.781 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:24 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1400, in _create_daemon 2024-09-21T12:34:24.781 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:24 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: out, err, code = await self._run_cephadm( 2024-09-21T12:34:24.781 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:24 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1743, in _run_cephadm 2024-09-21T12:34:24.781 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:24 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: raise OrchestratorError( 2024-09-21T12:34:24.781 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:24 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi079.vlhrek ... 2024-09-21T12:34:24.781 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:24 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: Creating ceph-iscsi config... 2024-09-21T12:34:24.781 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:24 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: Write file: /var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/iscsi.foo.smithi079.vlhrek/iscsi-gateway.cfg 2024-09-21T12:34:24.781 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:24 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: Write file: /var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/iscsi.foo.smithi079.vlhrek/tcmu-runner-entrypoint.sh 2024-09-21T12:34:24.781 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:24 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: Non-zero exit code 1 from systemctl restart ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek 2024-09-21T12:34:24.781 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:24 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: systemctl: stderr Job for ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service failed because the control process exited with error code. 2024-09-21T12:34:24.781 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:24 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: systemctl: stderr See "systemctl status ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service" and "journalctl -xeu ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service" for details. 2024-09-21T12:34:24.781 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:24 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: Traceback (most recent call last): 2024-09-21T12:34:24.781 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:24 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-21T12:34:24.781 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:24 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: return _run_code(code, main_globals, None, 2024-09-21T12:34:24.781 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:24 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-21T12:34:24.781 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:24 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: exec(code, run_globals) 2024-09-21T12:34:24.782 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:24 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 5628, in 2024-09-21T12:34:24.782 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:24 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 5616, in main 2024-09-21T12:34:24.782 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:24 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 3090, in command_deploy_from 2024-09-21T12:34:24.782 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:24 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 3125, in _common_deploy 2024-09-21T12:34:24.782 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:24 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 3145, in _deploy_daemon_container 2024-09-21T12:34:24.782 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:24 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 1147, in deploy_daemon 2024-09-21T12:34:24.782 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:24 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: File "/tmp/tmptpwpfqo2.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-21T12:34:24.782 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:24 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: RuntimeError: Failed command: systemctl restart ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek: Job for ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service failed because the control process exited with error code. 2024-09-21T12:34:24.782 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:24 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: See "systemctl status ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service" and "journalctl -xeu ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service" for details. 2024-09-21T12:34:25.644 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:25 smithi079 ceph-mon[23699]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi079.vlhrek ... 2024-09-21T12:34:25.645 INFO:journalctl@ceph.mon.a.smithi079.stdout: Creating ceph-iscsi config... 2024-09-21T12:34:25.645 INFO:journalctl@ceph.mon.a.smithi079.stdout: Write file: /var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/iscsi.foo.smithi079.vlhrek/iscsi-gateway.cfg 2024-09-21T12:34:25.645 INFO:journalctl@ceph.mon.a.smithi079.stdout: Write file: /var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/iscsi.foo.smithi079.vlhrek/tcmu-runner-entrypoint.sh 2024-09-21T12:34:25.645 INFO:journalctl@ceph.mon.a.smithi079.stdout: Non-zero exit code 1 from systemctl restart ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek 2024-09-21T12:34:25.645 INFO:journalctl@ceph.mon.a.smithi079.stdout: systemctl: stderr Job for ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service failed because the control process exited with error code. 2024-09-21T12:34:25.645 INFO:journalctl@ceph.mon.a.smithi079.stdout: systemctl: stderr See "systemctl status ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service" and "journalctl -xeu ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service" for details. 2024-09-21T12:34:25.645 INFO:journalctl@ceph.mon.a.smithi079.stdout: Traceback (most recent call last): 2024-09-21T12:34:25.645 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-21T12:34:25.645 INFO:journalctl@ceph.mon.a.smithi079.stdout: return _run_code(code, main_globals, None, 2024-09-21T12:34:25.645 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-21T12:34:25.645 INFO:journalctl@ceph.mon.a.smithi079.stdout: exec(code, run_globals) 2024-09-21T12:34:25.645 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 5628, in 2024-09-21T12:34:25.646 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 5616, in main 2024-09-21T12:34:25.646 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 3090, in command_deploy_from 2024-09-21T12:34:25.646 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 3125, in _common_deploy 2024-09-21T12:34:25.646 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 3145, in _deploy_daemon_container 2024-09-21T12:34:25.646 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 1147, in deploy_daemon 2024-09-21T12:34:25.646 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/tmp/tmptpwpfqo2.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-21T12:34:25.646 INFO:journalctl@ceph.mon.a.smithi079.stdout: RuntimeError: Failed command: systemctl restart ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek: Job for ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service failed because the control process exited with error code. 2024-09-21T12:34:25.646 INFO:journalctl@ceph.mon.a.smithi079.stdout: See "systemctl status ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service" and "journalctl -xeu ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service" for details. 2024-09-21T12:34:25.646 INFO:journalctl@ceph.mon.a.smithi079.stdout: Traceback (most recent call last): 2024-09-21T12:34:25.646 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1164, in _check_daemons 2024-09-21T12:34:25.646 INFO:journalctl@ceph.mon.a.smithi079.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-09-21T12:34:25.646 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2610, in _daemon_action 2024-09-21T12:34:25.646 INFO:journalctl@ceph.mon.a.smithi079.stdout: return self.wait_async( 2024-09-21T12:34:25.646 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 878, in wait_async 2024-09-21T12:34:25.647 INFO:journalctl@ceph.mon.a.smithi079.stdout: return self.event_loop.get_result(coro, timeout) 2024-09-21T12:34:25.647 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-09-21T12:34:25.647 INFO:journalctl@ceph.mon.a.smithi079.stdout: return future.result(timeout) 2024-09-21T12:34:25.647 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/lib64/python3.9/concurrent/futures/_base.py", line 446, in result 2024-09-21T12:34:25.647 INFO:journalctl@ceph.mon.a.smithi079.stdout: return self.__get_result() 2024-09-21T12:34:25.647 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/lib64/python3.9/concurrent/futures/_base.py", line 391, in __get_result 2024-09-21T12:34:25.647 INFO:journalctl@ceph.mon.a.smithi079.stdout: raise self._exception 2024-09-21T12:34:25.647 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1400, in _create_daemon 2024-09-21T12:34:25.647 INFO:journalctl@ceph.mon.a.smithi079.stdout: out, err, code = await self._run_cephadm( 2024-09-21T12:34:25.647 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1743, in _run_cephadm 2024-09-21T12:34:25.647 INFO:journalctl@ceph.mon.a.smithi079.stdout: raise OrchestratorError( 2024-09-21T12:34:25.647 INFO:journalctl@ceph.mon.a.smithi079.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi079.vlhrek ... 2024-09-21T12:34:25.647 INFO:journalctl@ceph.mon.a.smithi079.stdout: Creating ceph-iscsi config... 2024-09-21T12:34:25.647 INFO:journalctl@ceph.mon.a.smithi079.stdout: Write file: /var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/iscsi.foo.smithi079.vlhrek/iscsi-gateway.cfg 2024-09-21T12:34:25.648 INFO:journalctl@ceph.mon.a.smithi079.stdout: Write file: /var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/iscsi.foo.smithi079.vlhrek/tcmu-runner-entrypoint.sh 2024-09-21T12:34:25.648 INFO:journalctl@ceph.mon.a.smithi079.stdout: Non-zero exit code 1 from systemctl restart ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek 2024-09-21T12:34:25.648 INFO:journalctl@ceph.mon.a.smithi079.stdout: systemctl: stderr Job for ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service failed because the control process exited with error code. 2024-09-21T12:34:25.648 INFO:journalctl@ceph.mon.a.smithi079.stdout: systemctl: stderr See "systemctl status ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service" and "journalctl -xeu ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service" for details. 2024-09-21T12:34:25.648 INFO:journalctl@ceph.mon.a.smithi079.stdout: Traceback (most recent call last): 2024-09-21T12:34:25.648 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-21T12:34:25.648 INFO:journalctl@ceph.mon.a.smithi079.stdout: return _run_code(code, main_globals, None, 2024-09-21T12:34:25.648 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-21T12:34:25.648 INFO:journalctl@ceph.mon.a.smithi079.stdout: exec(code, run_globals) 2024-09-21T12:34:25.648 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 5628, in 2024-09-21T12:34:25.648 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 5616, in main 2024-09-21T12:34:25.648 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 3090, in command_deploy_from 2024-09-21T12:34:25.648 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 3125, in _common_deploy 2024-09-21T12:34:25.648 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 3145, in _deploy_daemon_container 2024-09-21T12:34:25.648 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 1147, in deploy_daemon 2024-09-21T12:34:25.649 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/tmp/tmptpwpfqo2.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-21T12:34:25.649 INFO:journalctl@ceph.mon.a.smithi079.stdout: RuntimeError: Failed command: systemctl restart ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek: Job for ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service failed because the control process exited with error code. 2024-09-21T12:34:25.649 INFO:journalctl@ceph.mon.a.smithi079.stdout: See "systemctl status ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service" and "journalctl -xeu ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service" for details. 2024-09-21T12:34:25.649 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:25 smithi079 ceph-mon[23699]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi104", "caps": []}]: dispatch 2024-09-21T12:34:25.649 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:25 smithi079 ceph-mon[30023]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi079.vlhrek ... 2024-09-21T12:34:25.649 INFO:journalctl@ceph.mon.c.smithi079.stdout: Creating ceph-iscsi config... 2024-09-21T12:34:25.649 INFO:journalctl@ceph.mon.c.smithi079.stdout: Write file: /var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/iscsi.foo.smithi079.vlhrek/iscsi-gateway.cfg 2024-09-21T12:34:25.649 INFO:journalctl@ceph.mon.c.smithi079.stdout: Write file: /var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/iscsi.foo.smithi079.vlhrek/tcmu-runner-entrypoint.sh 2024-09-21T12:34:25.649 INFO:journalctl@ceph.mon.c.smithi079.stdout: Non-zero exit code 1 from systemctl restart ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek 2024-09-21T12:34:25.649 INFO:journalctl@ceph.mon.c.smithi079.stdout: systemctl: stderr Job for ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service failed because the control process exited with error code. 2024-09-21T12:34:25.650 INFO:journalctl@ceph.mon.c.smithi079.stdout: systemctl: stderr See "systemctl status ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service" and "journalctl -xeu ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service" for details. 2024-09-21T12:34:25.650 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2024-09-21T12:34:25.650 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-21T12:34:25.650 INFO:journalctl@ceph.mon.c.smithi079.stdout: return _run_code(code, main_globals, None, 2024-09-21T12:34:25.650 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-21T12:34:25.650 INFO:journalctl@ceph.mon.c.smithi079.stdout: exec(code, run_globals) 2024-09-21T12:34:25.650 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 5628, in 2024-09-21T12:34:25.650 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 5616, in main 2024-09-21T12:34:25.650 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 3090, in command_deploy_from 2024-09-21T12:34:25.650 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 3125, in _common_deploy 2024-09-21T12:34:25.650 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 3145, in _deploy_daemon_container 2024-09-21T12:34:25.650 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 1147, in deploy_daemon 2024-09-21T12:34:25.650 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/tmp/tmptpwpfqo2.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-21T12:34:25.650 INFO:journalctl@ceph.mon.c.smithi079.stdout: RuntimeError: Failed command: systemctl restart ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek: Job for ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service failed because the control process exited with error code. 2024-09-21T12:34:25.650 INFO:journalctl@ceph.mon.c.smithi079.stdout: See "systemctl status ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service" and "journalctl -xeu ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service" for details. 2024-09-21T12:34:25.651 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2024-09-21T12:34:25.651 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1164, in _check_daemons 2024-09-21T12:34:25.651 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-09-21T12:34:25.651 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2610, in _daemon_action 2024-09-21T12:34:25.651 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.wait_async( 2024-09-21T12:34:25.651 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 878, in wait_async 2024-09-21T12:34:25.651 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro, timeout) 2024-09-21T12:34:25.651 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-09-21T12:34:25.651 INFO:journalctl@ceph.mon.c.smithi079.stdout: return future.result(timeout) 2024-09-21T12:34:25.651 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.9/concurrent/futures/_base.py", line 446, in result 2024-09-21T12:34:25.651 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2024-09-21T12:34:25.651 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.9/concurrent/futures/_base.py", line 391, in __get_result 2024-09-21T12:34:25.651 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2024-09-21T12:34:25.651 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1400, in _create_daemon 2024-09-21T12:34:25.651 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._run_cephadm( 2024-09-21T12:34:25.652 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1743, in _run_cephadm 2024-09-21T12:34:25.652 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError( 2024-09-21T12:34:25.652 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi079.vlhrek ... 2024-09-21T12:34:25.652 INFO:journalctl@ceph.mon.c.smithi079.stdout: Creating ceph-iscsi config... 2024-09-21T12:34:25.652 INFO:journalctl@ceph.mon.c.smithi079.stdout: Write file: /var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/iscsi.foo.smithi079.vlhrek/iscsi-gateway.cfg 2024-09-21T12:34:25.652 INFO:journalctl@ceph.mon.c.smithi079.stdout: Write file: /var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/iscsi.foo.smithi079.vlhrek/tcmu-runner-entrypoint.sh 2024-09-21T12:34:25.652 INFO:journalctl@ceph.mon.c.smithi079.stdout: Non-zero exit code 1 from systemctl restart ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek 2024-09-21T12:34:25.652 INFO:journalctl@ceph.mon.c.smithi079.stdout: systemctl: stderr Job for ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service failed because the control process exited with error code. 2024-09-21T12:34:25.652 INFO:journalctl@ceph.mon.c.smithi079.stdout: systemctl: stderr See "systemctl status ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service" and "journalctl -xeu ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service" for details. 2024-09-21T12:34:25.652 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2024-09-21T12:34:25.652 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-21T12:34:25.652 INFO:journalctl@ceph.mon.c.smithi079.stdout: return _run_code(code, main_globals, None, 2024-09-21T12:34:25.652 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-21T12:34:25.652 INFO:journalctl@ceph.mon.c.smithi079.stdout: exec(code, run_globals) 2024-09-21T12:34:25.652 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 5628, in 2024-09-21T12:34:25.653 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 5616, in main 2024-09-21T12:34:25.653 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 3090, in command_deploy_from 2024-09-21T12:34:25.653 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 3125, in _common_deploy 2024-09-21T12:34:25.653 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 3145, in _deploy_daemon_container 2024-09-21T12:34:25.653 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 1147, in deploy_daemon 2024-09-21T12:34:25.653 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/tmp/tmptpwpfqo2.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-21T12:34:25.653 INFO:journalctl@ceph.mon.c.smithi079.stdout: RuntimeError: Failed command: systemctl restart ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek: Job for ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service failed because the control process exited with error code. 2024-09-21T12:34:25.653 INFO:journalctl@ceph.mon.c.smithi079.stdout: See "systemctl status ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service" and "journalctl -xeu ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service" for details. 2024-09-21T12:34:25.653 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:25 smithi079 ceph-mon[30023]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi104", "caps": []}]: dispatch 2024-09-21T12:34:25.806 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:25 smithi104 ceph-mon[25639]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi079.vlhrek ... 2024-09-21T12:34:25.806 INFO:journalctl@ceph.mon.b.smithi104.stdout: Creating ceph-iscsi config... 2024-09-21T12:34:25.806 INFO:journalctl@ceph.mon.b.smithi104.stdout: Write file: /var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/iscsi.foo.smithi079.vlhrek/iscsi-gateway.cfg 2024-09-21T12:34:25.806 INFO:journalctl@ceph.mon.b.smithi104.stdout: Write file: /var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/iscsi.foo.smithi079.vlhrek/tcmu-runner-entrypoint.sh 2024-09-21T12:34:25.806 INFO:journalctl@ceph.mon.b.smithi104.stdout: Non-zero exit code 1 from systemctl restart ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek 2024-09-21T12:34:25.806 INFO:journalctl@ceph.mon.b.smithi104.stdout: systemctl: stderr Job for ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service failed because the control process exited with error code. 2024-09-21T12:34:25.807 INFO:journalctl@ceph.mon.b.smithi104.stdout: systemctl: stderr See "systemctl status ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service" and "journalctl -xeu ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service" for details. 2024-09-21T12:34:25.807 INFO:journalctl@ceph.mon.b.smithi104.stdout: Traceback (most recent call last): 2024-09-21T12:34:25.807 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-21T12:34:25.807 INFO:journalctl@ceph.mon.b.smithi104.stdout: return _run_code(code, main_globals, None, 2024-09-21T12:34:25.807 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-21T12:34:25.807 INFO:journalctl@ceph.mon.b.smithi104.stdout: exec(code, run_globals) 2024-09-21T12:34:25.807 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 5628, in 2024-09-21T12:34:25.807 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 5616, in main 2024-09-21T12:34:25.807 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 3090, in command_deploy_from 2024-09-21T12:34:25.807 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 3125, in _common_deploy 2024-09-21T12:34:25.807 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 3145, in _deploy_daemon_container 2024-09-21T12:34:25.807 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 1147, in deploy_daemon 2024-09-21T12:34:25.807 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/tmp/tmptpwpfqo2.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-21T12:34:25.808 INFO:journalctl@ceph.mon.b.smithi104.stdout: RuntimeError: Failed command: systemctl restart ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek: Job for ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service failed because the control process exited with error code. 2024-09-21T12:34:25.808 INFO:journalctl@ceph.mon.b.smithi104.stdout: See "systemctl status ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service" and "journalctl -xeu ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service" for details. 2024-09-21T12:34:25.808 INFO:journalctl@ceph.mon.b.smithi104.stdout: Traceback (most recent call last): 2024-09-21T12:34:25.808 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1164, in _check_daemons 2024-09-21T12:34:25.808 INFO:journalctl@ceph.mon.b.smithi104.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-09-21T12:34:25.808 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2610, in _daemon_action 2024-09-21T12:34:25.808 INFO:journalctl@ceph.mon.b.smithi104.stdout: return self.wait_async( 2024-09-21T12:34:25.808 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 878, in wait_async 2024-09-21T12:34:25.808 INFO:journalctl@ceph.mon.b.smithi104.stdout: return self.event_loop.get_result(coro, timeout) 2024-09-21T12:34:25.808 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-09-21T12:34:25.808 INFO:journalctl@ceph.mon.b.smithi104.stdout: return future.result(timeout) 2024-09-21T12:34:25.808 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/lib64/python3.9/concurrent/futures/_base.py", line 446, in result 2024-09-21T12:34:25.808 INFO:journalctl@ceph.mon.b.smithi104.stdout: return self.__get_result() 2024-09-21T12:34:25.808 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/lib64/python3.9/concurrent/futures/_base.py", line 391, in __get_result 2024-09-21T12:34:25.808 INFO:journalctl@ceph.mon.b.smithi104.stdout: raise self._exception 2024-09-21T12:34:25.809 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1400, in _create_daemon 2024-09-21T12:34:25.809 INFO:journalctl@ceph.mon.b.smithi104.stdout: out, err, code = await self._run_cephadm( 2024-09-21T12:34:25.809 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1743, in _run_cephadm 2024-09-21T12:34:25.809 INFO:journalctl@ceph.mon.b.smithi104.stdout: raise OrchestratorError( 2024-09-21T12:34:25.809 INFO:journalctl@ceph.mon.b.smithi104.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi079.vlhrek ... 2024-09-21T12:34:25.809 INFO:journalctl@ceph.mon.b.smithi104.stdout: Creating ceph-iscsi config... 2024-09-21T12:34:25.809 INFO:journalctl@ceph.mon.b.smithi104.stdout: Write file: /var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/iscsi.foo.smithi079.vlhrek/iscsi-gateway.cfg 2024-09-21T12:34:25.809 INFO:journalctl@ceph.mon.b.smithi104.stdout: Write file: /var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/iscsi.foo.smithi079.vlhrek/tcmu-runner-entrypoint.sh 2024-09-21T12:34:25.809 INFO:journalctl@ceph.mon.b.smithi104.stdout: Non-zero exit code 1 from systemctl restart ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek 2024-09-21T12:34:25.809 INFO:journalctl@ceph.mon.b.smithi104.stdout: systemctl: stderr Job for ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service failed because the control process exited with error code. 2024-09-21T12:34:25.809 INFO:journalctl@ceph.mon.b.smithi104.stdout: systemctl: stderr See "systemctl status ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service" and "journalctl -xeu ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service" for details. 2024-09-21T12:34:25.809 INFO:journalctl@ceph.mon.b.smithi104.stdout: Traceback (most recent call last): 2024-09-21T12:34:25.809 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-21T12:34:25.809 INFO:journalctl@ceph.mon.b.smithi104.stdout: return _run_code(code, main_globals, None, 2024-09-21T12:34:25.810 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-21T12:34:25.810 INFO:journalctl@ceph.mon.b.smithi104.stdout: exec(code, run_globals) 2024-09-21T12:34:25.810 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 5628, in 2024-09-21T12:34:25.810 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 5616, in main 2024-09-21T12:34:25.810 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 3090, in command_deploy_from 2024-09-21T12:34:25.810 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 3125, in _common_deploy 2024-09-21T12:34:25.810 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 3145, in _deploy_daemon_container 2024-09-21T12:34:25.810 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 1147, in deploy_daemon 2024-09-21T12:34:25.810 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/tmp/tmptpwpfqo2.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-21T12:34:25.810 INFO:journalctl@ceph.mon.b.smithi104.stdout: RuntimeError: Failed command: systemctl restart ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek: Job for ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service failed because the control process exited with error code. 2024-09-21T12:34:25.810 INFO:journalctl@ceph.mon.b.smithi104.stdout: See "systemctl status ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service" and "journalctl -xeu ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service" for details. 2024-09-21T12:34:25.810 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:25 smithi104 ceph-mon[25639]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi104", "caps": []}]: dispatch 2024-09-21T12:34:26.595 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:26 smithi079 ceph-mon[23699]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 16 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T12:34:26.595 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:26 smithi079 ceph-mon[23699]: Deploying daemon agent.smithi104 on smithi104 2024-09-21T12:34:26.595 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:26 smithi079 ceph-mon[23699]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' 2024-09-21T12:34:26.595 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:26 smithi079 ceph-mon[23699]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' 2024-09-21T12:34:26.595 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:26 smithi079 ceph-mon[30023]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 16 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T12:34:26.595 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:26 smithi079 ceph-mon[30023]: Deploying daemon agent.smithi104 on smithi104 2024-09-21T12:34:26.595 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:26 smithi079 ceph-mon[30023]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' 2024-09-21T12:34:26.595 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:26 smithi079 ceph-mon[30023]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' 2024-09-21T12:34:26.676 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:26 smithi104 ceph-mon[25639]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 16 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-21T12:34:26.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:26 smithi104 ceph-mon[25639]: Deploying daemon agent.smithi104 on smithi104 2024-09-21T12:34:26.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:26 smithi104 ceph-mon[25639]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' 2024-09-21T12:34:26.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:26 smithi104 ceph-mon[25639]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' 2024-09-21T12:34:27.676 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:27 smithi104 ceph-mon[25639]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-21T12:34:27.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:27 smithi104 ceph-mon[25639]: Reconfiguring daemon prometheus.a on smithi104 2024-09-21T12:34:27.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:27 smithi104 ceph-mon[25639]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' 2024-09-21T12:34:27.832 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:27 smithi079 ceph-mon[23699]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-21T12:34:27.833 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:27 smithi079 ceph-mon[23699]: Reconfiguring daemon prometheus.a on smithi104 2024-09-21T12:34:27.833 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:27 smithi079 ceph-mon[23699]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' 2024-09-21T12:34:27.833 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:27 smithi079 ceph-mon[30023]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-21T12:34:27.833 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:27 smithi079 ceph-mon[30023]: Reconfiguring daemon prometheus.a on smithi104 2024-09-21T12:34:27.833 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:27 smithi079 ceph-mon[30023]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' 2024-09-21T12:34:28.676 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:28 smithi104 ceph-mon[25639]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-09-21T12:34:28.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:28 smithi104 ceph-mon[25639]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' 2024-09-21T12:34:28.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:28 smithi104 ceph-mon[25639]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' 2024-09-21T12:34:28.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:28 smithi104 ceph-mon[25639]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' 2024-09-21T12:34:28.941 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:28 smithi079 ceph-mon[23699]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-09-21T12:34:28.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:28 smithi079 ceph-mon[23699]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' 2024-09-21T12:34:28.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:28 smithi079 ceph-mon[23699]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' 2024-09-21T12:34:28.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:28 smithi079 ceph-mon[23699]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' 2024-09-21T12:34:28.942 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:28 smithi079 ceph-mon[30023]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-09-21T12:34:28.942 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:28 smithi079 ceph-mon[30023]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' 2024-09-21T12:34:28.942 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:28 smithi079 ceph-mon[30023]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' 2024-09-21T12:34:28.942 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:28 smithi079 ceph-mon[30023]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' 2024-09-21T12:34:29.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:29 smithi104 ceph-mon[25639]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 748 B/s rd, 0 op/s 2024-09-21T12:34:29.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:29 smithi104 ceph-mon[25639]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-09-21T12:34:29.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:29 smithi079 ceph-mon[23699]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 748 B/s rd, 0 op/s 2024-09-21T12:34:29.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:29 smithi079 ceph-mon[23699]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-09-21T12:34:29.942 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:29 smithi079 ceph-mon[30023]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 748 B/s rd, 0 op/s 2024-09-21T12:34:29.942 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:29 smithi079 ceph-mon[30023]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-09-21T12:34:30.927 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:34:30 smithi104 systemd[1]: Stopping Ceph prometheus.a for 33168022-7814-11ef-baf6-efdc52797490... 2024-09-21T12:34:31.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:31 smithi104 ceph-mon[25639]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 280 B/s rd, 0 op/s 2024-09-21T12:34:31.927 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:34:31 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[49181]: ts=2024-09-21T12:34:31.752Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-09-21T12:34:31.927 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:34:31 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[49181]: ts=2024-09-21T12:34:31.752Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-09-21T12:34:31.927 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:34:31 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[49181]: ts=2024-09-21T12:34:31.752Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-09-21T12:34:31.927 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:34:31 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[49181]: ts=2024-09-21T12:34:31.752Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-09-21T12:34:31.927 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:34:31 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[49181]: ts=2024-09-21T12:34:31.752Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-09-21T12:34:31.927 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:34:31 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[49181]: ts=2024-09-21T12:34:31.752Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-09-21T12:34:31.927 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:34:31 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[49181]: ts=2024-09-21T12:34:31.752Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-09-21T12:34:31.927 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:34:31 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[49181]: ts=2024-09-21T12:34:31.752Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-09-21T12:34:31.928 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:34:31 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[49181]: ts=2024-09-21T12:34:31.752Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-09-21T12:34:31.928 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:34:31 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[49181]: ts=2024-09-21T12:34:31.753Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-09-21T12:34:31.928 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:34:31 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[49181]: ts=2024-09-21T12:34:31.753Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-09-21T12:34:31.928 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:34:31 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[49181]: ts=2024-09-21T12:34:31.753Z caller=main.go:1147 level=info msg="See you next time!" 2024-09-21T12:34:31.928 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:34:31 smithi104 podman[59244]: 2024-09-21 12:34:31.762764847 +0000 UTC m=+0.110106723 container died b307869246eeb9deb64c060fa0aa22ece21a2a71d0b274636b9778826bb98ecb (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T12:34:31.941 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:31 smithi079 ceph-mon[23699]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 280 B/s rd, 0 op/s 2024-09-21T12:34:31.942 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:31 smithi079 ceph-mon[30023]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 280 B/s rd, 0 op/s 2024-09-21T12:34:32.304 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:34:32 smithi104 podman[59244]: 2024-09-21 12:34:32.096790733 +0000 UTC m=+0.444132599 container cleanup b307869246eeb9deb64c060fa0aa22ece21a2a71d0b274636b9778826bb98ecb (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T12:34:32.306 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:34:32 smithi104 bash[59244]: ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a 2024-09-21T12:34:32.676 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:34:32 smithi104 podman[59279]: 2024-09-21 12:34:32.303380051 +0000 UTC m=+0.542214445 container remove b307869246eeb9deb64c060fa0aa22ece21a2a71d0b274636b9778826bb98ecb (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T12:34:32.677 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:34:32 smithi104 systemd[1]: ceph-33168022-7814-11ef-baf6-efdc52797490@prometheus.a.service: Deactivated successfully. 2024-09-21T12:34:32.677 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:34:32 smithi104 systemd[1]: Stopped Ceph prometheus.a for 33168022-7814-11ef-baf6-efdc52797490. 2024-09-21T12:34:32.677 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:34:32 smithi104 systemd[1]: ceph-33168022-7814-11ef-baf6-efdc52797490@prometheus.a.service: Consumed 1.842s CPU time. 2024-09-21T12:34:32.677 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:34:32 smithi104 systemd[1]: Starting Ceph prometheus.a for 33168022-7814-11ef-baf6-efdc52797490... 2024-09-21T12:34:33.024 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:34:32 smithi104 podman[59440]: 2024-09-21 12:34:32.674791612 +0000 UTC m=+0.017540304 image pull a07b618ecd1dce142bce4c52f0e80982eaf1f14265a2415c2d35978ccaa0a464 quay.io/prometheus/prometheus:v2.43.0 2024-09-21T12:34:33.024 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:34:32 smithi104 podman[59440]: 2024-09-21 12:34:32.779447473 +0000 UTC m=+0.122196142 container create bc3d876e1babfdb2598661dd161e5f63a4c06d3b789495ac6b920f4639321e1f (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T12:34:33.301 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:34:33 smithi104 podman[59440]: 2024-09-21 12:34:33.023497241 +0000 UTC m=+0.366245918 container init bc3d876e1babfdb2598661dd161e5f63a4c06d3b789495ac6b920f4639321e1f (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T12:34:33.301 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:34:33 smithi104 podman[59440]: 2024-09-21 12:34:33.027007604 +0000 UTC m=+0.369756275 container start bc3d876e1babfdb2598661dd161e5f63a4c06d3b789495ac6b920f4639321e1f (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T12:34:33.301 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:34:33 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[59471]: ts=2024-09-21T12:34:33.051Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-09-21T12:34:33.301 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:34:33 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[59471]: ts=2024-09-21T12:34:33.051Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-09-21T12:34:33.302 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:34:33 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[59471]: ts=2024-09-21T12:34:33.052Z caller=main.go:570 level=info host_details="(Linux 5.14.0-508.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024 x86_64 smithi104 (none))" 2024-09-21T12:34:33.302 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:34:33 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[59471]: ts=2024-09-21T12:34:33.052Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-09-21T12:34:33.302 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:34:33 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[59471]: ts=2024-09-21T12:34:33.052Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-09-21T12:34:33.302 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:34:33 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[59471]: ts=2024-09-21T12:34:33.053Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-09-21T12:34:33.302 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:34:33 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[59471]: ts=2024-09-21T12:34:33.053Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-09-21T12:34:33.302 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:34:33 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[59471]: ts=2024-09-21T12:34:33.055Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-09-21T12:34:33.302 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:34:33 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[59471]: ts=2024-09-21T12:34:33.056Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-09-21T12:34:33.302 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:34:33 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[59471]: ts=2024-09-21T12:34:33.057Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-09-21T12:34:33.302 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:34:33 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[59471]: ts=2024-09-21T12:34:33.057Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=2.446µs 2024-09-21T12:34:33.302 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:34:33 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[59471]: ts=2024-09-21T12:34:33.057Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-09-21T12:34:33.302 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:34:33 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[59471]: ts=2024-09-21T12:34:33.058Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=2 2024-09-21T12:34:33.302 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:34:33 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[59471]: ts=2024-09-21T12:34:33.068Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=2 2024-09-21T12:34:33.302 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:34:33 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[59471]: ts=2024-09-21T12:34:33.068Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=2 maxSegment=2 2024-09-21T12:34:33.303 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:34:33 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[59471]: ts=2024-09-21T12:34:33.068Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=34.953µs wal_replay_duration=10.967231ms wbl_replay_duration=169ns total_replay_duration=11.033051ms 2024-09-21T12:34:33.303 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:34:33 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[59471]: ts=2024-09-21T12:34:33.070Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-09-21T12:34:33.303 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:34:33 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[59471]: ts=2024-09-21T12:34:33.070Z caller=main.go:1029 level=info msg="TSDB started" 2024-09-21T12:34:33.303 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:34:33 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[59471]: ts=2024-09-21T12:34:33.070Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-09-21T12:34:33.303 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:34:33 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[59471]: ts=2024-09-21T12:34:33.095Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=24.545663ms db_storage=934ns remote_storage=1.416µs web_handler=429ns query_engine=633ns scrape=560.358µs scrape_sd=206.067µs notify=27.01µs notify_sd=16.744µs rules=23.153655ms tracing=6.387µs 2024-09-21T12:34:33.303 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:34:33 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[59471]: ts=2024-09-21T12:34:33.095Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-09-21T12:34:33.303 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:34:33 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[59471]: ts=2024-09-21T12:34:33.095Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-09-21T12:34:33.303 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:34:33 smithi104 bash[59440]: bc3d876e1babfdb2598661dd161e5f63a4c06d3b789495ac6b920f4639321e1f 2024-09-21T12:34:33.303 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:34:33 smithi104 systemd[1]: Started Ceph prometheus.a for 33168022-7814-11ef-baf6-efdc52797490. 2024-09-21T12:34:33.578 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:33 smithi104 ceph-mon[25639]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 280 B/s rd, 0 op/s 2024-09-21T12:34:33.579 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:33 smithi104 ceph-mon[25639]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' 2024-09-21T12:34:33.579 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:33 smithi104 ceph-mon[25639]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' 2024-09-21T12:34:33.579 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:33 smithi104 ceph-mon[25639]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-21T12:34:33.579 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:33 smithi104 ceph-mon[25639]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:34:33.860 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:33 smithi079 ceph-mon[23699]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 280 B/s rd, 0 op/s 2024-09-21T12:34:33.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:33 smithi079 ceph-mon[23699]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' 2024-09-21T12:34:33.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:33 smithi079 ceph-mon[23699]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' 2024-09-21T12:34:33.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:33 smithi079 ceph-mon[23699]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-21T12:34:33.861 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:33 smithi079 ceph-mon[23699]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:34:33.861 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:33 smithi079 ceph-mon[30023]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 280 B/s rd, 0 op/s 2024-09-21T12:34:33.861 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:33 smithi079 ceph-mon[30023]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' 2024-09-21T12:34:33.861 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:33 smithi079 ceph-mon[30023]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' 2024-09-21T12:34:33.861 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:33 smithi079 ceph-mon[30023]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-21T12:34:33.861 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:33 smithi079 ceph-mon[30023]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:34:34.855 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:34 smithi079 ceph-mon[23699]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-21T12:34:34.855 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:34 smithi079 ceph-mon[23699]: Upgrade: Need to upgrade myself (mgr.x) 2024-09-21T12:34:34.855 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:34 smithi079 ceph-mon[30023]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-21T12:34:34.855 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:34 smithi079 ceph-mon[30023]: Upgrade: Need to upgrade myself (mgr.x) 2024-09-21T12:34:34.896 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:34 smithi104 ceph-mon[25639]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-21T12:34:34.896 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:34 smithi104 ceph-mon[25639]: Upgrade: Need to upgrade myself (mgr.x) 2024-09-21T12:34:35.924 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:35 smithi079 ceph-mon[23699]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 280 B/s rd, 0 op/s 2024-09-21T12:34:35.925 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:35 smithi079 ceph-mon[23699]: Upgrade: Updating mgr.y 2024-09-21T12:34:35.925 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:35 smithi079 ceph-mon[23699]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' 2024-09-21T12:34:35.925 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:35 smithi079 ceph-mon[23699]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-21T12:34:35.925 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:35 smithi079 ceph-mon[23699]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-21T12:34:35.925 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:35 smithi079 ceph-mon[23699]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:34:35.925 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:35 smithi079 ceph-mon[23699]: Deploying daemon mgr.y on smithi079 2024-09-21T12:34:35.925 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:35 smithi079 ceph-mon[23699]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' 2024-09-21T12:34:35.925 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:35 smithi079 ceph-mon[30023]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 280 B/s rd, 0 op/s 2024-09-21T12:34:35.925 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:35 smithi079 ceph-mon[30023]: Upgrade: Updating mgr.y 2024-09-21T12:34:35.925 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:35 smithi079 ceph-mon[30023]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' 2024-09-21T12:34:35.925 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:35 smithi079 ceph-mon[30023]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-21T12:34:35.925 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:35 smithi079 ceph-mon[30023]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-21T12:34:35.926 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:35 smithi079 ceph-mon[30023]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:34:35.926 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:35 smithi079 ceph-mon[30023]: Deploying daemon mgr.y on smithi079 2024-09-21T12:34:35.926 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:35 smithi079 ceph-mon[30023]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' 2024-09-21T12:34:35.926 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:35 smithi104 ceph-mon[25639]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 280 B/s rd, 0 op/s 2024-09-21T12:34:35.926 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:35 smithi104 ceph-mon[25639]: Upgrade: Updating mgr.y 2024-09-21T12:34:35.926 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:35 smithi104 ceph-mon[25639]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' 2024-09-21T12:34:35.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:35 smithi104 ceph-mon[25639]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-21T12:34:35.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:35 smithi104 ceph-mon[25639]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-21T12:34:35.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:35 smithi104 ceph-mon[25639]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:34:35.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:35 smithi104 ceph-mon[25639]: Deploying daemon mgr.y on smithi079 2024-09-21T12:34:35.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:35 smithi104 ceph-mon[25639]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' 2024-09-21T12:34:36.581 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:34:36 smithi079 systemd[1]: Stopping Ceph mgr.y for 33168022-7814-11ef-baf6-efdc52797490... 2024-09-21T12:34:36.831 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:36 smithi079 ceph-mon[23699]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' 2024-09-21T12:34:36.831 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:36 smithi079 ceph-mon[23699]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' 2024-09-21T12:34:36.831 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:36 smithi079 ceph-mon[23699]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:34:36.831 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:36 smithi079 ceph-mon[23699]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:34:36.831 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:36 smithi079 ceph-mon[23699]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' 2024-09-21T12:34:36.832 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:36 smithi079 ceph-mon[30023]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' 2024-09-21T12:34:36.832 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:36 smithi079 ceph-mon[30023]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' 2024-09-21T12:34:36.832 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:36 smithi079 ceph-mon[30023]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:34:36.832 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:36 smithi079 ceph-mon[30023]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:34:36.832 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:36 smithi079 ceph-mon[30023]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' 2024-09-21T12:34:36.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:36 smithi104 ceph-mon[25639]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' 2024-09-21T12:34:36.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:36 smithi104 ceph-mon[25639]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' 2024-09-21T12:34:36.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:36 smithi104 ceph-mon[25639]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:34:36.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:36 smithi104 ceph-mon[25639]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:34:36.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:36 smithi104 ceph-mon[25639]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' 2024-09-21T12:34:37.191 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:34:37 smithi079 podman[70352]: 2024-09-21 12:34:37.029322894 +0000 UTC m=+0.155367823 container died e34c6cf8a73d28c0780d22cef3c05911324af321edfdd4de57c877e59e1e16ad (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=reef-6a1bdf8, org.label-schema.build-date=20240731, ceph=True, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-09-21T12:34:37.627 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:34:37 smithi079 podman[70352]: 2024-09-21 12:34:37.456858808 +0000 UTC m=+0.582903737 container cleanup e34c6cf8a73d28c0780d22cef3c05911324af321edfdd4de57c877e59e1e16ad (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=reef-6a1bdf8, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-09-21T12:34:37.628 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:34:37 smithi079 bash[70352]: ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y 2024-09-21T12:34:37.885 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:37 smithi079 ceph-mon[30023]: Health check update: 2 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-09-21T12:34:37.885 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:37 smithi079 ceph-mon[30023]: from='client.? 172.21.15.79:0/729663977' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-21T12:34:37.885 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:37 smithi079 ceph-mon[30023]: from='client.? 172.21.15.79:0/2912199013' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:0/3835416297"}]: dispatch 2024-09-21T12:34:37.885 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:37 smithi079 ceph-mon[23699]: Health check update: 2 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-09-21T12:34:37.885 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:37 smithi079 ceph-mon[23699]: from='client.? 172.21.15.79:0/729663977' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-21T12:34:37.885 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:37 smithi079 ceph-mon[23699]: from='client.? 172.21.15.79:0/2912199013' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:0/3835416297"}]: dispatch 2024-09-21T12:34:37.886 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:34:37 smithi079 podman[70413]: 2024-09-21 12:34:37.791881826 +0000 UTC m=+0.758640734 container remove e34c6cf8a73d28c0780d22cef3c05911324af321edfdd4de57c877e59e1e16ad (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=reef-6a1bdf8, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T12:34:37.886 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:34:37 smithi079 systemd[1]: ceph-33168022-7814-11ef-baf6-efdc52797490@mgr.y.service: Main process exited, code=exited, status=143/n/a 2024-09-21T12:34:38.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:37 smithi104 ceph-mon[25639]: Health check update: 2 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-09-21T12:34:38.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:37 smithi104 ceph-mon[25639]: from='client.? 172.21.15.79:0/729663977' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-21T12:34:38.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:37 smithi104 ceph-mon[25639]: from='client.? 172.21.15.79:0/2912199013' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:0/3835416297"}]: dispatch 2024-09-21T12:34:38.191 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:34:38 smithi079 systemd[1]: ceph-33168022-7814-11ef-baf6-efdc52797490@mgr.y.service: Failed with result 'exit-code'. 2024-09-21T12:34:38.192 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:34:38 smithi079 systemd[1]: Stopped Ceph mgr.y for 33168022-7814-11ef-baf6-efdc52797490. 2024-09-21T12:34:38.192 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:34:38 smithi079 systemd[1]: ceph-33168022-7814-11ef-baf6-efdc52797490@mgr.y.service: Consumed 1min 1.734s CPU time. 2024-09-21T12:34:38.513 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:34:38 smithi079 systemd[1]: Starting Ceph mgr.y for 33168022-7814-11ef-baf6-efdc52797490... 2024-09-21T12:34:38.908 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:38 smithi079 ceph-mon[30023]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:34:38.908 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:38 smithi079 ceph-mon[30023]: from='client.? 172.21.15.79:0/2912199013' entity='client.iscsi.foo.smithi079.vlhrek' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:0/3835416297"}]': finished 2024-09-21T12:34:38.908 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:38 smithi079 ceph-mon[30023]: osdmap e97: 8 total, 8 up, 8 in 2024-09-21T12:34:38.908 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:38 smithi079 ceph-mon[30023]: from='client.? 172.21.15.79:0/2456985762' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:0/2336253920"}]: dispatch 2024-09-21T12:34:38.909 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:38 smithi079 ceph-mon[23699]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:34:38.909 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:38 smithi079 ceph-mon[23699]: from='client.? 172.21.15.79:0/2912199013' entity='client.iscsi.foo.smithi079.vlhrek' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:0/3835416297"}]': finished 2024-09-21T12:34:38.909 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:38 smithi079 ceph-mon[23699]: osdmap e97: 8 total, 8 up, 8 in 2024-09-21T12:34:38.909 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:38 smithi079 ceph-mon[23699]: from='client.? 172.21.15.79:0/2456985762' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:0/2336253920"}]: dispatch 2024-09-21T12:34:39.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:38 smithi104 ceph-mon[25639]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:34:39.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:38 smithi104 ceph-mon[25639]: from='client.? 172.21.15.79:0/2912199013' entity='client.iscsi.foo.smithi079.vlhrek' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:0/3835416297"}]': finished 2024-09-21T12:34:39.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:38 smithi104 ceph-mon[25639]: osdmap e97: 8 total, 8 up, 8 in 2024-09-21T12:34:39.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:38 smithi104 ceph-mon[25639]: from='client.? 172.21.15.79:0/2456985762' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:0/2336253920"}]: dispatch 2024-09-21T12:34:39.942 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:39 smithi079 ceph-mon[30023]: from='client.? 172.21.15.79:0/2456985762' entity='client.iscsi.foo.smithi079.vlhrek' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:0/2336253920"}]': finished 2024-09-21T12:34:39.942 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:39 smithi079 ceph-mon[30023]: osdmap e98: 8 total, 8 up, 8 in 2024-09-21T12:34:39.942 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:39 smithi079 ceph-mon[30023]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' 2024-09-21T12:34:39.942 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:39 smithi079 ceph-mon[30023]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T12:34:39.942 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:39 smithi079 ceph-mon[30023]: from='client.? 172.21.15.79:0/3676448211' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:6801/2765998793"}]: dispatch 2024-09-21T12:34:39.942 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:39 smithi079 ceph-mon[30023]: from='client.? ' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:6801/2765998793"}]: dispatch 2024-09-21T12:34:39.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:39 smithi079 ceph-mon[23699]: from='client.? 172.21.15.79:0/2456985762' entity='client.iscsi.foo.smithi079.vlhrek' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:0/2336253920"}]': finished 2024-09-21T12:34:39.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:39 smithi079 ceph-mon[23699]: osdmap e98: 8 total, 8 up, 8 in 2024-09-21T12:34:39.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:39 smithi079 ceph-mon[23699]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' 2024-09-21T12:34:39.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:39 smithi079 ceph-mon[23699]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T12:34:39.943 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:39 smithi079 ceph-mon[23699]: from='client.? 172.21.15.79:0/3676448211' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:6801/2765998793"}]: dispatch 2024-09-21T12:34:39.943 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:39 smithi079 ceph-mon[23699]: from='client.? ' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:6801/2765998793"}]: dispatch 2024-09-21T12:34:40.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:39 smithi104 ceph-mon[25639]: from='client.? 172.21.15.79:0/2456985762' entity='client.iscsi.foo.smithi079.vlhrek' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:0/2336253920"}]': finished 2024-09-21T12:34:40.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:39 smithi104 ceph-mon[25639]: osdmap e98: 8 total, 8 up, 8 in 2024-09-21T12:34:40.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:39 smithi104 ceph-mon[25639]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' 2024-09-21T12:34:40.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:39 smithi104 ceph-mon[25639]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T12:34:40.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:39 smithi104 ceph-mon[25639]: from='client.? 172.21.15.79:0/3676448211' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:6801/2765998793"}]: dispatch 2024-09-21T12:34:40.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:39 smithi104 ceph-mon[25639]: from='client.? ' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:6801/2765998793"}]: dispatch 2024-09-21T12:34:40.442 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:34:40 smithi079 podman[70825]: 2024-09-21 12:34:40.064155015 +0000 UTC m=+0.345732897 image pull b415816db8b41f544cdfdc71d193ff8b1ee008bf7b0ec4f64da8eb8f71b62e85 quay.ceph.io/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5 2024-09-21T12:34:40.442 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:34:40 smithi079 podman[70825]: 2024-09-21 12:34:40.16204797 +0000 UTC m=+0.443625850 container create b1744e3101ac1b54b1b4340f1666ce2a050e7a87cf0f3c5958dcd304e082ed63 (image=quay.ceph.io/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-main-2024-09-21-0842-4417849, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731) 2024-09-21T12:34:40.442 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:34:40 smithi079 podman[70825]: 2024-09-21 12:34:40.364512481 +0000 UTC m=+0.646090371 container init b1744e3101ac1b54b1b4340f1666ce2a050e7a87cf0f3c5958dcd304e082ed63 (image=quay.ceph.io/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y, ceph=True, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-guits-main-2024-09-21-0842-4417849, org.label-schema.license=GPLv2) 2024-09-21T12:34:40.442 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:34:40 smithi079 podman[70825]: 2024-09-21 12:34:40.367544814 +0000 UTC m=+0.649122696 container start b1744e3101ac1b54b1b4340f1666ce2a050e7a87cf0f3c5958dcd304e082ed63 (image=quay.ceph.io/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, RELEASE=wip-guits-main-2024-09-21-0842-4417849, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0) 2024-09-21T12:34:40.442 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:34:40 smithi079 ceph-mgr[70887]: -- 172.21.15.79:0/660632175 <== mon.0 v2:172.21.15.79:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x558add65d0e0 con 0x558add63b400 2024-09-21T12:34:40.785 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:40 smithi079 ceph-mon[23699]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 174 B/s rd, 0 op/s 2024-09-21T12:34:40.785 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:40 smithi079 ceph-mon[23699]: from='client.? ' entity='client.iscsi.foo.smithi079.vlhrek' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:6801/2765998793"}]': finished 2024-09-21T12:34:40.785 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:40 smithi079 ceph-mon[23699]: osdmap e99: 8 total, 8 up, 8 in 2024-09-21T12:34:40.785 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:40 smithi079 ceph-mon[23699]: from='client.? 172.21.15.79:0/2093803470' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:0/390660218"}]: dispatch 2024-09-21T12:34:40.785 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:40 smithi079 ceph-mon[23699]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' 2024-09-21T12:34:40.785 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:40 smithi079 ceph-mon[23699]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' 2024-09-21T12:34:40.785 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:40 smithi079 ceph-mon[23699]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:34:40.786 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:40 smithi079 ceph-mon[30023]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 174 B/s rd, 0 op/s 2024-09-21T12:34:40.786 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:40 smithi079 ceph-mon[30023]: from='client.? ' entity='client.iscsi.foo.smithi079.vlhrek' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:6801/2765998793"}]': finished 2024-09-21T12:34:40.786 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:40 smithi079 ceph-mon[30023]: osdmap e99: 8 total, 8 up, 8 in 2024-09-21T12:34:40.786 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:40 smithi079 ceph-mon[30023]: from='client.? 172.21.15.79:0/2093803470' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:0/390660218"}]: dispatch 2024-09-21T12:34:40.786 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:40 smithi079 ceph-mon[30023]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' 2024-09-21T12:34:40.786 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:40 smithi079 ceph-mon[30023]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' 2024-09-21T12:34:40.786 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:40 smithi079 ceph-mon[30023]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:34:40.786 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:34:40 smithi079 bash[70825]: b1744e3101ac1b54b1b4340f1666ce2a050e7a87cf0f3c5958dcd304e082ed63 2024-09-21T12:34:40.786 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:34:40 smithi079 systemd[1]: Started Ceph mgr.y for 33168022-7814-11ef-baf6-efdc52797490. 2024-09-21T12:34:41.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:40 smithi104 ceph-mon[25639]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 174 B/s rd, 0 op/s 2024-09-21T12:34:41.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:40 smithi104 ceph-mon[25639]: from='client.? ' entity='client.iscsi.foo.smithi079.vlhrek' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:6801/2765998793"}]': finished 2024-09-21T12:34:41.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:40 smithi104 ceph-mon[25639]: osdmap e99: 8 total, 8 up, 8 in 2024-09-21T12:34:41.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:40 smithi104 ceph-mon[25639]: from='client.? 172.21.15.79:0/2093803470' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:0/390660218"}]: dispatch 2024-09-21T12:34:41.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:40 smithi104 ceph-mon[25639]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' 2024-09-21T12:34:41.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:40 smithi104 ceph-mon[25639]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' 2024-09-21T12:34:41.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:40 smithi104 ceph-mon[25639]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:34:41.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:41 smithi079 ceph-mon[23699]: from='client.? 172.21.15.79:0/2093803470' entity='client.iscsi.foo.smithi079.vlhrek' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:0/390660218"}]': finished 2024-09-21T12:34:41.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:41 smithi079 ceph-mon[23699]: osdmap e100: 8 total, 8 up, 8 in 2024-09-21T12:34:41.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:41 smithi079 ceph-mon[23699]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' 2024-09-21T12:34:41.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:41 smithi079 ceph-mon[23699]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' 2024-09-21T12:34:41.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:41 smithi079 ceph-mon[23699]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:34:41.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:41 smithi079 ceph-mon[23699]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:34:41.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:41 smithi079 ceph-mon[23699]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 324 B/s rd, 0 op/s 2024-09-21T12:34:41.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:41 smithi079 ceph-mon[23699]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 324 B/s rd, 0 op/s 2024-09-21T12:34:41.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:41 smithi079 ceph-mon[23699]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' 2024-09-21T12:34:41.943 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:41 smithi079 ceph-mon[23699]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-21T12:34:41.943 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:41 smithi079 ceph-mon[23699]: Reconfiguring iscsi.foo.smithi079.vlhrek (dependencies changed)... 2024-09-21T12:34:41.943 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:41 smithi079 ceph-mon[23699]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi079.vlhrek", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-21T12:34:41.943 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:41 smithi079 ceph-mon[23699]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:34:41.943 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:41 smithi079 ceph-mon[23699]: Reconfiguring daemon iscsi.foo.smithi079.vlhrek on smithi079 2024-09-21T12:34:41.943 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:41 smithi079 ceph-mon[23699]: from='client.? 172.21.15.79:0/1795629423' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:6800/2765998793"}]: dispatch 2024-09-21T12:34:41.943 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:41 smithi079 ceph-mon[23699]: from='client.? ' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:6800/2765998793"}]: dispatch 2024-09-21T12:34:41.943 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:41 smithi079 ceph-mon[30023]: from='client.? 172.21.15.79:0/2093803470' entity='client.iscsi.foo.smithi079.vlhrek' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:0/390660218"}]': finished 2024-09-21T12:34:41.943 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:41 smithi079 ceph-mon[30023]: osdmap e100: 8 total, 8 up, 8 in 2024-09-21T12:34:41.943 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:41 smithi079 ceph-mon[30023]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' 2024-09-21T12:34:41.944 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:41 smithi079 ceph-mon[30023]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' 2024-09-21T12:34:41.944 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:41 smithi079 ceph-mon[30023]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:34:41.944 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:41 smithi079 ceph-mon[30023]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:34:41.944 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:41 smithi079 ceph-mon[30023]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 324 B/s rd, 0 op/s 2024-09-21T12:34:41.944 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:41 smithi079 ceph-mon[30023]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 324 B/s rd, 0 op/s 2024-09-21T12:34:41.944 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:41 smithi079 ceph-mon[30023]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' 2024-09-21T12:34:41.944 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:41 smithi079 ceph-mon[30023]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-21T12:34:41.944 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:41 smithi079 ceph-mon[30023]: Reconfiguring iscsi.foo.smithi079.vlhrek (dependencies changed)... 2024-09-21T12:34:41.944 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:41 smithi079 ceph-mon[30023]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi079.vlhrek", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-21T12:34:41.944 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:41 smithi079 ceph-mon[30023]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:34:41.944 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:41 smithi079 ceph-mon[30023]: Reconfiguring daemon iscsi.foo.smithi079.vlhrek on smithi079 2024-09-21T12:34:41.944 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:41 smithi079 ceph-mon[30023]: from='client.? 172.21.15.79:0/1795629423' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:6800/2765998793"}]: dispatch 2024-09-21T12:34:41.944 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:41 smithi079 ceph-mon[30023]: from='client.? ' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:6800/2765998793"}]: dispatch 2024-09-21T12:34:42.177 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:41 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: ::ffff:172.21.15.104 - - [21/Sep/2024:12:34:41] "GET /metrics HTTP/1.1" 200 38586 "" "Prometheus/2.43.0" 2024-09-21T12:34:42.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:41 smithi104 ceph-mon[25639]: from='client.? 172.21.15.79:0/2093803470' entity='client.iscsi.foo.smithi079.vlhrek' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:0/390660218"}]': finished 2024-09-21T12:34:42.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:41 smithi104 ceph-mon[25639]: osdmap e100: 8 total, 8 up, 8 in 2024-09-21T12:34:42.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:41 smithi104 ceph-mon[25639]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' 2024-09-21T12:34:42.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:41 smithi104 ceph-mon[25639]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' 2024-09-21T12:34:42.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:41 smithi104 ceph-mon[25639]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:34:42.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:41 smithi104 ceph-mon[25639]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:34:42.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:41 smithi104 ceph-mon[25639]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 324 B/s rd, 0 op/s 2024-09-21T12:34:42.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:41 smithi104 ceph-mon[25639]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 324 B/s rd, 0 op/s 2024-09-21T12:34:42.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:41 smithi104 ceph-mon[25639]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' 2024-09-21T12:34:42.178 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:41 smithi104 ceph-mon[25639]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-21T12:34:42.178 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:41 smithi104 ceph-mon[25639]: Reconfiguring iscsi.foo.smithi079.vlhrek (dependencies changed)... 2024-09-21T12:34:42.178 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:41 smithi104 ceph-mon[25639]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi079.vlhrek", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-21T12:34:42.178 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:41 smithi104 ceph-mon[25639]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:34:42.178 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:41 smithi104 ceph-mon[25639]: Reconfiguring daemon iscsi.foo.smithi079.vlhrek on smithi079 2024-09-21T12:34:42.178 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:41 smithi104 ceph-mon[25639]: from='client.? 172.21.15.79:0/1795629423' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:6800/2765998793"}]: dispatch 2024-09-21T12:34:42.178 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:41 smithi104 ceph-mon[25639]: from='client.? ' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:6800/2765998793"}]: dispatch 2024-09-21T12:34:43.176 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:42 smithi104 ceph-mon[25639]: from='client.? ' entity='client.iscsi.foo.smithi079.vlhrek' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:6800/2765998793"}]': finished 2024-09-21T12:34:43.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:42 smithi104 ceph-mon[25639]: osdmap e101: 8 total, 8 up, 8 in 2024-09-21T12:34:43.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:42 smithi104 ceph-mon[25639]: from='client.? 172.21.15.79:0/3100708034' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:0/3698254455"}]: dispatch 2024-09-21T12:34:43.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:42 smithi104 ceph-mon[25639]: from='client.? ' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:0/3698254455"}]: dispatch 2024-09-21T12:34:43.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:42 smithi079 ceph-mon[23699]: from='client.? ' entity='client.iscsi.foo.smithi079.vlhrek' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:6800/2765998793"}]': finished 2024-09-21T12:34:43.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:42 smithi079 ceph-mon[23699]: osdmap e101: 8 total, 8 up, 8 in 2024-09-21T12:34:43.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:42 smithi079 ceph-mon[23699]: from='client.? 172.21.15.79:0/3100708034' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:0/3698254455"}]: dispatch 2024-09-21T12:34:43.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:42 smithi079 ceph-mon[23699]: from='client.? ' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:0/3698254455"}]: dispatch 2024-09-21T12:34:43.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:42 smithi079 ceph-mon[30023]: from='client.? ' entity='client.iscsi.foo.smithi079.vlhrek' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:6800/2765998793"}]': finished 2024-09-21T12:34:43.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:42 smithi079 ceph-mon[30023]: osdmap e101: 8 total, 8 up, 8 in 2024-09-21T12:34:43.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:42 smithi079 ceph-mon[30023]: from='client.? 172.21.15.79:0/3100708034' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:0/3698254455"}]: dispatch 2024-09-21T12:34:43.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:42 smithi079 ceph-mon[30023]: from='client.? ' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:0/3698254455"}]: dispatch 2024-09-21T12:34:44.176 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:43 smithi104 ceph-mon[25639]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:34:44.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:43 smithi104 ceph-mon[25639]: from='client.? ' entity='client.iscsi.foo.smithi079.vlhrek' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:0/3698254455"}]': finished 2024-09-21T12:34:44.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:43 smithi104 ceph-mon[25639]: osdmap e102: 8 total, 8 up, 8 in 2024-09-21T12:34:44.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:43 smithi104 ceph-mon[25639]: from='client.? 172.21.15.79:0/1403648195' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:0/1760224796"}]: dispatch 2024-09-21T12:34:44.191 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:43 smithi079 ceph-mon[23699]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:34:44.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:43 smithi079 ceph-mon[23699]: from='client.? ' entity='client.iscsi.foo.smithi079.vlhrek' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:0/3698254455"}]': finished 2024-09-21T12:34:44.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:43 smithi079 ceph-mon[23699]: osdmap e102: 8 total, 8 up, 8 in 2024-09-21T12:34:44.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:43 smithi079 ceph-mon[23699]: from='client.? 172.21.15.79:0/1403648195' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:0/1760224796"}]: dispatch 2024-09-21T12:34:44.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:43 smithi079 ceph-mon[30023]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:34:44.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:43 smithi079 ceph-mon[30023]: from='client.? ' entity='client.iscsi.foo.smithi079.vlhrek' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:0/3698254455"}]': finished 2024-09-21T12:34:44.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:43 smithi079 ceph-mon[30023]: osdmap e102: 8 total, 8 up, 8 in 2024-09-21T12:34:44.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:43 smithi079 ceph-mon[30023]: from='client.? 172.21.15.79:0/1403648195' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:0/1760224796"}]: dispatch 2024-09-21T12:34:45.176 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:44 smithi104 ceph-mon[25639]: from='client.? 172.21.15.79:0/1403648195' entity='client.iscsi.foo.smithi079.vlhrek' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:0/1760224796"}]': finished 2024-09-21T12:34:45.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:44 smithi104 ceph-mon[25639]: osdmap e103: 8 total, 8 up, 8 in 2024-09-21T12:34:45.191 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:44 smithi079 ceph-mon[23699]: from='client.? 172.21.15.79:0/1403648195' entity='client.iscsi.foo.smithi079.vlhrek' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:0/1760224796"}]': finished 2024-09-21T12:34:45.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:44 smithi079 ceph-mon[23699]: osdmap e103: 8 total, 8 up, 8 in 2024-09-21T12:34:45.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:44 smithi079 ceph-mon[30023]: from='client.? 172.21.15.79:0/1403648195' entity='client.iscsi.foo.smithi079.vlhrek' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:0/1760224796"}]': finished 2024-09-21T12:34:45.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:44 smithi079 ceph-mon[30023]: osdmap e103: 8 total, 8 up, 8 in 2024-09-21T12:34:45.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:45 smithi079 ceph-mon[23699]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:34:45.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:45 smithi079 ceph-mon[23699]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' 2024-09-21T12:34:45.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:45 smithi079 ceph-mon[23699]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' 2024-09-21T12:34:45.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:45 smithi079 ceph-mon[23699]: Standby manager daemon y restarted 2024-09-21T12:34:45.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:45 smithi079 ceph-mon[23699]: Standby manager daemon y started 2024-09-21T12:34:45.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:45 smithi079 ceph-mon[23699]: from='mgr.? 172.21.15.79:0/1857156883' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-21T12:34:45.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:45 smithi079 ceph-mon[23699]: from='mgr.? 172.21.15.79:0/1857156883' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-21T12:34:45.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:45 smithi079 ceph-mon[23699]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' 2024-09-21T12:34:45.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:45 smithi079 ceph-mon[23699]: from='mgr.? 172.21.15.79:0/1857156883' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-09-21T12:34:45.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:45 smithi079 ceph-mon[23699]: from='mgr.? 172.21.15.79:0/1857156883' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-21T12:34:45.943 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:34:45 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[70883]: [21/Sep/2024:12:34:45] ENGINE Bus STARTING 2024-09-21T12:34:45.943 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:34:45 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[70883]: CherryPy Checker: 2024-09-21T12:34:45.943 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:34:45 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[70883]: The Application mounted at '' has an empty config. 2024-09-21T12:34:45.943 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:34:45 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[70883]: 2024-09-21T12:34:45.943 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:45 smithi079 ceph-mon[30023]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:34:45.943 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:45 smithi079 ceph-mon[30023]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' 2024-09-21T12:34:45.943 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:45 smithi079 ceph-mon[30023]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' 2024-09-21T12:34:45.943 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:45 smithi079 ceph-mon[30023]: Standby manager daemon y restarted 2024-09-21T12:34:45.943 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:45 smithi079 ceph-mon[30023]: Standby manager daemon y started 2024-09-21T12:34:45.943 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:45 smithi079 ceph-mon[30023]: from='mgr.? 172.21.15.79:0/1857156883' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-21T12:34:45.943 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:45 smithi079 ceph-mon[30023]: from='mgr.? 172.21.15.79:0/1857156883' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-21T12:34:45.944 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:45 smithi079 ceph-mon[30023]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' 2024-09-21T12:34:45.944 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:45 smithi079 ceph-mon[30023]: from='mgr.? 172.21.15.79:0/1857156883' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-09-21T12:34:45.944 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:45 smithi079 ceph-mon[30023]: from='mgr.? 172.21.15.79:0/1857156883' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-21T12:34:46.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:45 smithi104 ceph-mon[25639]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:34:46.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:45 smithi104 ceph-mon[25639]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' 2024-09-21T12:34:46.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:45 smithi104 ceph-mon[25639]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' 2024-09-21T12:34:46.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:45 smithi104 ceph-mon[25639]: Standby manager daemon y restarted 2024-09-21T12:34:46.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:45 smithi104 ceph-mon[25639]: Standby manager daemon y started 2024-09-21T12:34:46.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:45 smithi104 ceph-mon[25639]: from='mgr.? 172.21.15.79:0/1857156883' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-21T12:34:46.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:45 smithi104 ceph-mon[25639]: from='mgr.? 172.21.15.79:0/1857156883' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-21T12:34:46.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:45 smithi104 ceph-mon[25639]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' 2024-09-21T12:34:46.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:45 smithi104 ceph-mon[25639]: from='mgr.? 172.21.15.79:0/1857156883' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-09-21T12:34:46.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:45 smithi104 ceph-mon[25639]: from='mgr.? 172.21.15.79:0/1857156883' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-21T12:34:46.354 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:34:45 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[70883]: [21/Sep/2024:12:34:45] ENGINE Serving on http://:::9283 2024-09-21T12:34:46.355 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:34:45 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[70883]: [21/Sep/2024:12:34:45] ENGINE Bus STARTED 2024-09-21T12:34:46.844 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:46 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: 2024-09-21T12:34:46.838+0000 7fb3ad11f640 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi079.vlhrek ... 2024-09-21T12:34:46.844 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:46 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: Creating ceph-iscsi config... 2024-09-21T12:34:46.844 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:46 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: Write file: /var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/iscsi.foo.smithi079.vlhrek/iscsi-gateway.cfg 2024-09-21T12:34:46.844 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:46 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: Write file: /var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/iscsi.foo.smithi079.vlhrek/tcmu-runner-entrypoint.sh 2024-09-21T12:34:46.844 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:46 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: Non-zero exit code 1 from systemctl restart ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek 2024-09-21T12:34:46.845 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:46 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: systemctl: stderr Job for ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service failed because the control process exited with error code. 2024-09-21T12:34:46.845 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:46 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: systemctl: stderr See "systemctl status ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service" and "journalctl -xeu ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service" for details. 2024-09-21T12:34:46.845 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:46 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: Traceback (most recent call last): 2024-09-21T12:34:46.845 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:46 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-21T12:34:46.845 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:46 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: return _run_code(code, main_globals, None, 2024-09-21T12:34:46.845 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:46 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-21T12:34:46.845 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:46 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: exec(code, run_globals) 2024-09-21T12:34:46.845 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:46 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 5628, in 2024-09-21T12:34:46.845 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:46 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 5616, in main 2024-09-21T12:34:46.845 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:46 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 3090, in command_deploy_from 2024-09-21T12:34:46.845 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:46 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 3125, in _common_deploy 2024-09-21T12:34:46.845 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:46 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 3145, in _deploy_daemon_container 2024-09-21T12:34:46.845 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:46 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 1147, in deploy_daemon 2024-09-21T12:34:46.845 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:46 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: File "/tmp/tmptpwpfqo2.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-21T12:34:46.846 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:46 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: RuntimeError: Failed command: systemctl restart ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek: Job for ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service failed because the control process exited with error code. 2024-09-21T12:34:46.846 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:46 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: See "systemctl status ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service" and "journalctl -xeu ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service" for details. 2024-09-21T12:34:46.846 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:46 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: Traceback (most recent call last): 2024-09-21T12:34:46.846 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:46 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1164, in _check_daemons 2024-09-21T12:34:46.846 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:46 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: self.mgr._daemon_action(daemon_spec, action=action) 2024-09-21T12:34:46.846 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:46 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2610, in _daemon_action 2024-09-21T12:34:46.846 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:46 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: return self.wait_async( 2024-09-21T12:34:46.847 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:46 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: File "/usr/share/ceph/mgr/cephadm/module.py", line 878, in wait_async 2024-09-21T12:34:46.847 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:46 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: return self.event_loop.get_result(coro, timeout) 2024-09-21T12:34:46.847 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:46 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-09-21T12:34:46.847 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:46 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: return future.result(timeout) 2024-09-21T12:34:46.847 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:46 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: File "/lib64/python3.9/concurrent/futures/_base.py", line 446, in result 2024-09-21T12:34:46.847 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:46 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: return self.__get_result() 2024-09-21T12:34:46.847 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:46 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: File "/lib64/python3.9/concurrent/futures/_base.py", line 391, in __get_result 2024-09-21T12:34:46.847 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:46 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: raise self._exception 2024-09-21T12:34:46.847 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:46 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1400, in _create_daemon 2024-09-21T12:34:46.847 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:46 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: out, err, code = await self._run_cephadm( 2024-09-21T12:34:46.847 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:46 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1743, in _run_cephadm 2024-09-21T12:34:46.847 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:46 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: raise OrchestratorError( 2024-09-21T12:34:46.847 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:46 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi079.vlhrek ... 2024-09-21T12:34:46.847 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:46 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: Creating ceph-iscsi config... 2024-09-21T12:34:46.847 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:46 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: Write file: /var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/iscsi.foo.smithi079.vlhrek/iscsi-gateway.cfg 2024-09-21T12:34:46.848 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:46 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: Write file: /var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/iscsi.foo.smithi079.vlhrek/tcmu-runner-entrypoint.sh 2024-09-21T12:34:46.848 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:46 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: Non-zero exit code 1 from systemctl restart ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek 2024-09-21T12:34:46.848 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:46 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: systemctl: stderr Job for ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service failed because the control process exited with error code. 2024-09-21T12:34:46.848 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:46 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: systemctl: stderr See "systemctl status ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service" and "journalctl -xeu ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service" for details. 2024-09-21T12:34:46.848 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:46 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: Traceback (most recent call last): 2024-09-21T12:34:46.848 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:46 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-21T12:34:46.848 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:46 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: return _run_code(code, main_globals, None, 2024-09-21T12:34:46.848 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:46 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-21T12:34:46.848 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:46 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: exec(code, run_globals) 2024-09-21T12:34:46.848 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:46 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 5628, in 2024-09-21T12:34:46.848 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:46 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 5616, in main 2024-09-21T12:34:46.848 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:46 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 3090, in command_deploy_from 2024-09-21T12:34:46.848 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:46 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 3125, in _common_deploy 2024-09-21T12:34:46.848 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:46 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 3145, in _deploy_daemon_container 2024-09-21T12:34:46.849 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:46 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 1147, in deploy_daemon 2024-09-21T12:34:46.849 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:46 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: File "/tmp/tmptpwpfqo2.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-21T12:34:46.849 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:46 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: RuntimeError: Failed command: systemctl restart ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek: Job for ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service failed because the control process exited with error code. 2024-09-21T12:34:46.849 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:46 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: See "systemctl status ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service" and "journalctl -xeu ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service" for details. 2024-09-21T12:34:46.926 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:46 smithi104 ceph-mon[25639]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1021 B/s rd, 0 op/s 2024-09-21T12:34:46.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:46 smithi104 ceph-mon[25639]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:34:46.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:46 smithi104 ceph-mon[25639]: Health check update: 3 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-09-21T12:34:46.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:46 smithi104 ceph-mon[25639]: mgrmap e27: x(active, since 39s), standbys: y 2024-09-21T12:34:47.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:46 smithi079 ceph-mon[23699]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1021 B/s rd, 0 op/s 2024-09-21T12:34:47.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:46 smithi079 ceph-mon[23699]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:34:47.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:46 smithi079 ceph-mon[23699]: Health check update: 3 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-09-21T12:34:47.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:46 smithi079 ceph-mon[23699]: mgrmap e27: x(active, since 39s), standbys: y 2024-09-21T12:34:47.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:46 smithi079 ceph-mon[30023]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1021 B/s rd, 0 op/s 2024-09-21T12:34:47.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:46 smithi079 ceph-mon[30023]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T12:34:47.193 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:46 smithi079 ceph-mon[30023]: Health check update: 3 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-09-21T12:34:47.193 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:46 smithi079 ceph-mon[30023]: mgrmap e27: x(active, since 39s), standbys: y 2024-09-21T12:34:47.193 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:34:46 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[70883]: [21/Sep/2024:12:34:46] ENGINE Bus STOPPING 2024-09-21T12:34:47.193 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:34:46 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[70883]: [21/Sep/2024:12:34:46] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-21T12:34:47.193 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:34:46 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[70883]: [21/Sep/2024:12:34:46] ENGINE Bus STOPPED 2024-09-21T12:34:47.304 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:46 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: ignoring --setuser ceph since I am not root 2024-09-21T12:34:47.304 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:46 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: ignoring --setgroup ceph since I am not root 2024-09-21T12:34:47.304 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:46 smithi104 ceph-mgr[56352]: -- 172.21.15.104:0/1480946295 <== mon.1 v2:172.21.15.104:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55fbe0e270e0 con 0x55fbe0e04800 2024-09-21T12:34:48.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:47 smithi104 ceph-mon[25639]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:34:48.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:47 smithi104 ceph-mon[25639]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-09-21T12:34:48.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:47 smithi104 ceph-mon[25639]: Activating manager daemon y 2024-09-21T12:34:48.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:47 smithi104 ceph-mon[25639]: osdmap e104: 8 total, 8 up, 8 in 2024-09-21T12:34:48.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:47 smithi104 ceph-mon[25639]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2024-09-21T12:34:48.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:47 smithi104 ceph-mon[25639]: mgrmap e28: y(active, starting, since 0.00847609s) 2024-09-21T12:34:48.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:47 smithi104 ceph-mon[25639]: from='mgr.15351 172.21.15.79:0/1857156883' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T12:34:48.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:47 smithi104 ceph-mon[25639]: from='mgr.15351 172.21.15.79:0/1857156883' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T12:34:48.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:47 smithi104 ceph-mon[25639]: from='mgr.15351 172.21.15.79:0/1857156883' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T12:34:48.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:47 smithi104 ceph-mon[25639]: from='mgr.15351 172.21.15.79:0/1857156883' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-21T12:34:48.178 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:47 smithi104 ceph-mon[25639]: from='mgr.15351 172.21.15.79:0/1857156883' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T12:34:48.178 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:47 smithi104 ceph-mon[25639]: from='mgr.15351 172.21.15.79:0/1857156883' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T12:34:48.178 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:47 smithi104 ceph-mon[25639]: from='mgr.15351 172.21.15.79:0/1857156883' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T12:34:48.178 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:47 smithi104 ceph-mon[25639]: from='mgr.15351 172.21.15.79:0/1857156883' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T12:34:48.178 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:47 smithi104 ceph-mon[25639]: from='mgr.15351 172.21.15.79:0/1857156883' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T12:34:48.178 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:47 smithi104 ceph-mon[25639]: from='mgr.15351 172.21.15.79:0/1857156883' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T12:34:48.178 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:47 smithi104 ceph-mon[25639]: from='mgr.15351 172.21.15.79:0/1857156883' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T12:34:48.178 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:47 smithi104 ceph-mon[25639]: from='mgr.15351 172.21.15.79:0/1857156883' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T12:34:48.178 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:47 smithi104 ceph-mon[25639]: from='mgr.15351 172.21.15.79:0/1857156883' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-21T12:34:48.178 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:47 smithi104 ceph-mon[25639]: from='mgr.15351 172.21.15.79:0/1857156883' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-21T12:34:48.178 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:47 smithi104 ceph-mon[25639]: from='mgr.15351 172.21.15.79:0/1857156883' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-21T12:34:48.178 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:47 smithi104 ceph-mon[25639]: Manager daemon y is now available 2024-09-21T12:34:48.178 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:47 smithi104 ceph-mon[25639]: from='mgr.15351 ' entity='mgr.y' 2024-09-21T12:34:48.178 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:47 smithi104 ceph-mon[25639]: from='mgr.15351 172.21.15.79:0/1857156883' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:34:48.179 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:47 smithi104 ceph-mon[25639]: from='mgr.15351 172.21.15.79:0/1857156883' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi079", "caps": []}]: dispatch 2024-09-21T12:34:48.179 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:47 smithi104 ceph-mon[25639]: from='mgr.15351 172.21.15.79:0/1857156883' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi104", "caps": []}]: dispatch 2024-09-21T12:34:48.179 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:47 smithi104 ceph-mon[25639]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi079", "caps": []}]: dispatch 2024-09-21T12:34:48.179 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:47 smithi104 ceph-mon[25639]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi104", "caps": []}]: dispatch 2024-09-21T12:34:48.179 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:47 smithi104 ceph-mon[25639]: from='mgr.15351 172.21.15.79:0/1857156883' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T12:34:48.179 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:47 smithi104 ceph-mon[25639]: from='mgr.15351 172.21.15.79:0/1857156883' entity='mgr.y' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T12:34:48.179 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:47 smithi104 ceph-mon[25639]: from='mgr.15351 172.21.15.79:0/1857156883' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-21T12:34:48.179 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:47 smithi104 ceph-mon[25639]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-21T12:34:48.179 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:47 smithi104 ceph-mon[25639]: from='mgr.15351 172.21.15.79:0/1857156883' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-21T12:34:48.179 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:47 smithi104 ceph-mon[25639]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-21T12:34:48.179 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:47 smithi104 ceph-mon[25639]: from='mgr.15351 172.21.15.79:0/1857156883' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi104", "name": "osd_memory_target"}]: dispatch 2024-09-21T12:34:48.179 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:47 smithi104 ceph-mon[25639]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi104", "name": "osd_memory_target"}]: dispatch 2024-09-21T12:34:48.192 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:34:47 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[70883]: [21/Sep/2024:12:34:47] ENGINE Bus STARTING 2024-09-21T12:34:48.192 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:34:48 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[70883]: [21/Sep/2024:12:34:48] ENGINE Serving on http://:::9283 2024-09-21T12:34:48.192 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:34:48 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[70883]: [21/Sep/2024:12:34:48] ENGINE Bus STARTED 2024-09-21T12:34:48.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:47 smithi079 ceph-mon[30023]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:34:48.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:47 smithi079 ceph-mon[30023]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-09-21T12:34:48.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:47 smithi079 ceph-mon[30023]: Activating manager daemon y 2024-09-21T12:34:48.193 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:47 smithi079 ceph-mon[30023]: osdmap e104: 8 total, 8 up, 8 in 2024-09-21T12:34:48.193 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:47 smithi079 ceph-mon[30023]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2024-09-21T12:34:48.193 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:47 smithi079 ceph-mon[30023]: mgrmap e28: y(active, starting, since 0.00847609s) 2024-09-21T12:34:48.193 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:47 smithi079 ceph-mon[30023]: from='mgr.15351 172.21.15.79:0/1857156883' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T12:34:48.193 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:47 smithi079 ceph-mon[30023]: from='mgr.15351 172.21.15.79:0/1857156883' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T12:34:48.193 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:47 smithi079 ceph-mon[30023]: from='mgr.15351 172.21.15.79:0/1857156883' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T12:34:48.193 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:47 smithi079 ceph-mon[30023]: from='mgr.15351 172.21.15.79:0/1857156883' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-21T12:34:48.193 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:47 smithi079 ceph-mon[30023]: from='mgr.15351 172.21.15.79:0/1857156883' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T12:34:48.193 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:47 smithi079 ceph-mon[30023]: from='mgr.15351 172.21.15.79:0/1857156883' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T12:34:48.193 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:47 smithi079 ceph-mon[30023]: from='mgr.15351 172.21.15.79:0/1857156883' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T12:34:48.193 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:47 smithi079 ceph-mon[30023]: from='mgr.15351 172.21.15.79:0/1857156883' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T12:34:48.193 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:47 smithi079 ceph-mon[30023]: from='mgr.15351 172.21.15.79:0/1857156883' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T12:34:48.193 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:47 smithi079 ceph-mon[30023]: from='mgr.15351 172.21.15.79:0/1857156883' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T12:34:48.194 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:47 smithi079 ceph-mon[30023]: from='mgr.15351 172.21.15.79:0/1857156883' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T12:34:48.194 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:47 smithi079 ceph-mon[30023]: from='mgr.15351 172.21.15.79:0/1857156883' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T12:34:48.194 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:47 smithi079 ceph-mon[30023]: from='mgr.15351 172.21.15.79:0/1857156883' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-21T12:34:48.194 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:47 smithi079 ceph-mon[30023]: from='mgr.15351 172.21.15.79:0/1857156883' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-21T12:34:48.194 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:47 smithi079 ceph-mon[30023]: from='mgr.15351 172.21.15.79:0/1857156883' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-21T12:34:48.194 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:47 smithi079 ceph-mon[30023]: Manager daemon y is now available 2024-09-21T12:34:48.194 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:47 smithi079 ceph-mon[30023]: from='mgr.15351 ' entity='mgr.y' 2024-09-21T12:34:48.194 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:47 smithi079 ceph-mon[30023]: from='mgr.15351 172.21.15.79:0/1857156883' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:34:48.194 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:47 smithi079 ceph-mon[30023]: from='mgr.15351 172.21.15.79:0/1857156883' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi079", "caps": []}]: dispatch 2024-09-21T12:34:48.194 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:47 smithi079 ceph-mon[30023]: from='mgr.15351 172.21.15.79:0/1857156883' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi104", "caps": []}]: dispatch 2024-09-21T12:34:48.194 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:47 smithi079 ceph-mon[30023]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi079", "caps": []}]: dispatch 2024-09-21T12:34:48.194 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:47 smithi079 ceph-mon[30023]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi104", "caps": []}]: dispatch 2024-09-21T12:34:48.194 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:47 smithi079 ceph-mon[30023]: from='mgr.15351 172.21.15.79:0/1857156883' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T12:34:48.194 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:47 smithi079 ceph-mon[30023]: from='mgr.15351 172.21.15.79:0/1857156883' entity='mgr.y' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T12:34:48.195 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:47 smithi079 ceph-mon[30023]: from='mgr.15351 172.21.15.79:0/1857156883' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-21T12:34:48.195 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:47 smithi079 ceph-mon[30023]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-21T12:34:48.195 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:47 smithi079 ceph-mon[30023]: from='mgr.15351 172.21.15.79:0/1857156883' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-21T12:34:48.195 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:47 smithi079 ceph-mon[30023]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-21T12:34:48.195 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:47 smithi079 ceph-mon[30023]: from='mgr.15351 172.21.15.79:0/1857156883' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi104", "name": "osd_memory_target"}]: dispatch 2024-09-21T12:34:48.195 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:47 smithi079 ceph-mon[30023]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi104", "name": "osd_memory_target"}]: dispatch 2024-09-21T12:34:48.195 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:47 smithi079 ceph-mon[23699]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:34:48.195 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:47 smithi079 ceph-mon[23699]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-09-21T12:34:48.195 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:47 smithi079 ceph-mon[23699]: Activating manager daemon y 2024-09-21T12:34:48.195 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:47 smithi079 ceph-mon[23699]: osdmap e104: 8 total, 8 up, 8 in 2024-09-21T12:34:48.195 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:47 smithi079 ceph-mon[23699]: from='mgr.24949 172.21.15.104:0/1289373660' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2024-09-21T12:34:48.195 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:47 smithi079 ceph-mon[23699]: mgrmap e28: y(active, starting, since 0.00847609s) 2024-09-21T12:34:48.196 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:47 smithi079 ceph-mon[23699]: from='mgr.15351 172.21.15.79:0/1857156883' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T12:34:48.196 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:47 smithi079 ceph-mon[23699]: from='mgr.15351 172.21.15.79:0/1857156883' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T12:34:48.196 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:47 smithi079 ceph-mon[23699]: from='mgr.15351 172.21.15.79:0/1857156883' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T12:34:48.196 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:47 smithi079 ceph-mon[23699]: from='mgr.15351 172.21.15.79:0/1857156883' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-21T12:34:48.196 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:47 smithi079 ceph-mon[23699]: from='mgr.15351 172.21.15.79:0/1857156883' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T12:34:48.196 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:47 smithi079 ceph-mon[23699]: from='mgr.15351 172.21.15.79:0/1857156883' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T12:34:48.196 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:47 smithi079 ceph-mon[23699]: from='mgr.15351 172.21.15.79:0/1857156883' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T12:34:48.196 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:47 smithi079 ceph-mon[23699]: from='mgr.15351 172.21.15.79:0/1857156883' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T12:34:48.196 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:47 smithi079 ceph-mon[23699]: from='mgr.15351 172.21.15.79:0/1857156883' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T12:34:48.196 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:47 smithi079 ceph-mon[23699]: from='mgr.15351 172.21.15.79:0/1857156883' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T12:34:48.196 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:47 smithi079 ceph-mon[23699]: from='mgr.15351 172.21.15.79:0/1857156883' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T12:34:48.196 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:47 smithi079 ceph-mon[23699]: from='mgr.15351 172.21.15.79:0/1857156883' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T12:34:48.196 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:47 smithi079 ceph-mon[23699]: from='mgr.15351 172.21.15.79:0/1857156883' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-21T12:34:48.196 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:47 smithi079 ceph-mon[23699]: from='mgr.15351 172.21.15.79:0/1857156883' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-21T12:34:48.197 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:47 smithi079 ceph-mon[23699]: from='mgr.15351 172.21.15.79:0/1857156883' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-21T12:34:48.197 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:47 smithi079 ceph-mon[23699]: Manager daemon y is now available 2024-09-21T12:34:48.197 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:47 smithi079 ceph-mon[23699]: from='mgr.15351 ' entity='mgr.y' 2024-09-21T12:34:48.197 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:47 smithi079 ceph-mon[23699]: from='mgr.15351 172.21.15.79:0/1857156883' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:34:48.197 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:47 smithi079 ceph-mon[23699]: from='mgr.15351 172.21.15.79:0/1857156883' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi079", "caps": []}]: dispatch 2024-09-21T12:34:48.197 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:47 smithi079 ceph-mon[23699]: from='mgr.15351 172.21.15.79:0/1857156883' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi104", "caps": []}]: dispatch 2024-09-21T12:34:48.197 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:47 smithi079 ceph-mon[23699]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi079", "caps": []}]: dispatch 2024-09-21T12:34:48.197 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:47 smithi079 ceph-mon[23699]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi104", "caps": []}]: dispatch 2024-09-21T12:34:48.197 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:47 smithi079 ceph-mon[23699]: from='mgr.15351 172.21.15.79:0/1857156883' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T12:34:48.197 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:47 smithi079 ceph-mon[23699]: from='mgr.15351 172.21.15.79:0/1857156883' entity='mgr.y' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T12:34:48.197 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:47 smithi079 ceph-mon[23699]: from='mgr.15351 172.21.15.79:0/1857156883' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-21T12:34:48.197 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:47 smithi079 ceph-mon[23699]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-21T12:34:48.197 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:47 smithi079 ceph-mon[23699]: from='mgr.15351 172.21.15.79:0/1857156883' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-21T12:34:48.197 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:47 smithi079 ceph-mon[23699]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-21T12:34:48.197 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:47 smithi079 ceph-mon[23699]: from='mgr.15351 172.21.15.79:0/1857156883' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi104", "name": "osd_memory_target"}]: dispatch 2024-09-21T12:34:48.197 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:47 smithi079 ceph-mon[23699]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi104", "name": "osd_memory_target"}]: dispatch 2024-09-21T12:34:48.693 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:34:48 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[70883]: 2024-09-21T12:34:48.420+0000 7f82a0995640 -1 log_channel(cephadm) log [ERR] : [21/Sep/2024:12:34:48] ENGINE Error in HTTPServer.serve 2024-09-21T12:34:48.693 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:34:48 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[70883]: Traceback (most recent call last): 2024-09-21T12:34:48.694 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:34:48 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[70883]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T12:34:48.694 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:34:48 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[70883]: self._connections.run(self.expiration_interval) 2024-09-21T12:34:48.694 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:34:48 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[70883]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T12:34:48.694 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:34:48 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[70883]: self._run(expiration_interval) 2024-09-21T12:34:48.694 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:34:48 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[70883]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T12:34:48.694 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:34:48 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[70883]: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T12:34:48.694 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:34:48 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[70883]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T12:34:48.694 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:34:48 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[70883]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T12:34:48.694 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:34:48 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[70883]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T12:34:48.694 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:34:48 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[70883]: s = self.context.wrap_socket( 2024-09-21T12:34:48.694 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:34:48 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[70883]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T12:34:48.694 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:34:48 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[70883]: return self.sslsocket_class._create( 2024-09-21T12:34:48.694 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:34:48 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[70883]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T12:34:48.694 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:34:48 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[70883]: self.do_handshake() 2024-09-21T12:34:48.694 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:34:48 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[70883]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T12:34:48.695 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:34:48 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[70883]: self._sslobj.do_handshake() 2024-09-21T12:34:48.695 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:34:48 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[70883]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T12:34:48.695 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:34:48 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[70883]: 2024-09-21T12:34:49.107 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:48 smithi104 ceph-mon[25639]: mgrmap e29: y(active, since 1.00879s) 2024-09-21T12:34:49.107 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:48 smithi104 ceph-mon[25639]: [21/Sep/2024:12:34:48] ENGINE Bus STARTING 2024-09-21T12:34:49.107 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:48 smithi104 ceph-mon[25639]: [21/Sep/2024:12:34:48] ENGINE Serving on http://172.21.15.79:8765 2024-09-21T12:34:49.191 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:48 smithi079 ceph-mon[23699]: mgrmap e29: y(active, since 1.00879s) 2024-09-21T12:34:49.191 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:48 smithi079 ceph-mon[23699]: [21/Sep/2024:12:34:48] ENGINE Bus STARTING 2024-09-21T12:34:49.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:48 smithi079 ceph-mon[23699]: [21/Sep/2024:12:34:48] ENGINE Serving on http://172.21.15.79:8765 2024-09-21T12:34:49.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:48 smithi079 ceph-mon[30023]: mgrmap e29: y(active, since 1.00879s) 2024-09-21T12:34:49.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:48 smithi079 ceph-mon[30023]: [21/Sep/2024:12:34:48] ENGINE Bus STARTING 2024-09-21T12:34:49.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:48 smithi079 ceph-mon[30023]: [21/Sep/2024:12:34:48] ENGINE Serving on http://172.21.15.79:8765 2024-09-21T12:34:50.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:49 smithi104 ceph-mon[25639]: [21/Sep/2024:12:34:48] ENGINE Serving on https://172.21.15.79:7150 2024-09-21T12:34:50.178 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:49 smithi104 ceph-mon[25639]: [21/Sep/2024:12:34:48] ENGINE Bus STARTED 2024-09-21T12:34:50.178 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:49 smithi104 ceph-mon[25639]: [21/Sep/2024:12:34:48] ENGINE Error in HTTPServer.serve 2024-09-21T12:34:50.178 INFO:journalctl@ceph.mon.b.smithi104.stdout: Traceback (most recent call last): 2024-09-21T12:34:50.178 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T12:34:50.178 INFO:journalctl@ceph.mon.b.smithi104.stdout: self._connections.run(self.expiration_interval) 2024-09-21T12:34:50.178 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T12:34:50.178 INFO:journalctl@ceph.mon.b.smithi104.stdout: self._run(expiration_interval) 2024-09-21T12:34:50.178 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T12:34:50.178 INFO:journalctl@ceph.mon.b.smithi104.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T12:34:50.178 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T12:34:50.178 INFO:journalctl@ceph.mon.b.smithi104.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T12:34:50.179 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T12:34:50.179 INFO:journalctl@ceph.mon.b.smithi104.stdout: s = self.context.wrap_socket( 2024-09-21T12:34:50.179 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T12:34:50.179 INFO:journalctl@ceph.mon.b.smithi104.stdout: return self.sslsocket_class._create( 2024-09-21T12:34:50.179 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T12:34:50.179 INFO:journalctl@ceph.mon.b.smithi104.stdout: self.do_handshake() 2024-09-21T12:34:50.179 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T12:34:50.179 INFO:journalctl@ceph.mon.b.smithi104.stdout: self._sslobj.do_handshake() 2024-09-21T12:34:50.179 INFO:journalctl@ceph.mon.b.smithi104.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T12:34:50.179 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:49 smithi104 ceph-mon[25639]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 3 failed cephadm daemon(s)) 2024-09-21T12:34:50.179 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:49 smithi104 ceph-mon[25639]: Cluster is now healthy 2024-09-21T12:34:50.179 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:49 smithi104 ceph-mon[25639]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:34:50.179 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:49 smithi104 ceph-mon[25639]: from='mgr.15351 172.21.15.79:0/1857156883' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi079", "name": "osd_memory_target"}]: dispatch 2024-09-21T12:34:50.180 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:49 smithi104 ceph-mon[25639]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi079", "name": "osd_memory_target"}]: dispatch 2024-09-21T12:34:50.180 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:49 smithi104 ceph-mon[25639]: from='mgr.15351 172.21.15.79:0/1857156883' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:34:50.180 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:49 smithi104 ceph-mon[25639]: from='mgr.15351 172.21.15.79:0/1857156883' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:34:50.180 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:49 smithi104 ceph-mon[25639]: Updating smithi079:/etc/ceph/ceph.conf 2024-09-21T12:34:50.180 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:49 smithi104 ceph-mon[25639]: Updating smithi104:/etc/ceph/ceph.conf 2024-09-21T12:34:50.180 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:49 smithi104 ceph-mon[25639]: from='mgr.15351 ' entity='mgr.y' 2024-09-21T12:34:50.180 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:49 smithi104 ceph-mon[25639]: from='mgr.15351 ' entity='mgr.y' 2024-09-21T12:34:50.180 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:49 smithi104 ceph-mon[25639]: Updating smithi079:/var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T12:34:50.180 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:49 smithi104 ceph-mon[25639]: from='mgr.15351 ' entity='mgr.y' 2024-09-21T12:34:50.180 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:49 smithi104 ceph-mon[25639]: Updating smithi104:/var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T12:34:50.180 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:49 smithi104 ceph-mon[25639]: from='mgr.15351 ' entity='mgr.y' 2024-09-21T12:34:50.180 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:49 smithi104 ceph-mon[25639]: from='mgr.15351 ' entity='mgr.y' 2024-09-21T12:34:50.180 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:49 smithi104 ceph-mon[25639]: from='mgr.15351 ' entity='mgr.y' 2024-09-21T12:34:50.180 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:49 smithi104 ceph-mon[25639]: from='mgr.15351 ' entity='mgr.y' 2024-09-21T12:34:50.180 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:49 smithi104 ceph-mon[25639]: from='mgr.15351 ' entity='mgr.y' 2024-09-21T12:34:50.181 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:49 smithi104 ceph-mon[25639]: from='mgr.15351 ' entity='mgr.y' 2024-09-21T12:34:50.181 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:49 smithi104 ceph-mon[25639]: from='mgr.15351 ' entity='mgr.y' 2024-09-21T12:34:50.181 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:49 smithi104 ceph-mon[25639]: from='mgr.15351 ' entity='mgr.y' 2024-09-21T12:34:50.181 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:49 smithi104 ceph-mon[25639]: from='mgr.15351 ' entity='mgr.y' 2024-09-21T12:34:50.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:49 smithi079 ceph-mon[30023]: [21/Sep/2024:12:34:48] ENGINE Serving on https://172.21.15.79:7150 2024-09-21T12:34:50.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:49 smithi079 ceph-mon[30023]: [21/Sep/2024:12:34:48] ENGINE Bus STARTED 2024-09-21T12:34:50.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:49 smithi079 ceph-mon[30023]: [21/Sep/2024:12:34:48] ENGINE Error in HTTPServer.serve 2024-09-21T12:34:50.192 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2024-09-21T12:34:50.192 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T12:34:50.192 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._connections.run(self.expiration_interval) 2024-09-21T12:34:50.192 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T12:34:50.192 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._run(expiration_interval) 2024-09-21T12:34:50.192 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T12:34:50.192 INFO:journalctl@ceph.mon.c.smithi079.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T12:34:50.193 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T12:34:50.193 INFO:journalctl@ceph.mon.c.smithi079.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T12:34:50.193 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T12:34:50.193 INFO:journalctl@ceph.mon.c.smithi079.stdout: s = self.context.wrap_socket( 2024-09-21T12:34:50.193 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T12:34:50.193 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.sslsocket_class._create( 2024-09-21T12:34:50.193 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T12:34:50.193 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.do_handshake() 2024-09-21T12:34:50.193 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T12:34:50.193 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._sslobj.do_handshake() 2024-09-21T12:34:50.193 INFO:journalctl@ceph.mon.c.smithi079.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T12:34:50.193 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:49 smithi079 ceph-mon[30023]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 3 failed cephadm daemon(s)) 2024-09-21T12:34:50.193 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:49 smithi079 ceph-mon[30023]: Cluster is now healthy 2024-09-21T12:34:50.194 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:49 smithi079 ceph-mon[30023]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:34:50.194 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:49 smithi079 ceph-mon[30023]: from='mgr.15351 172.21.15.79:0/1857156883' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi079", "name": "osd_memory_target"}]: dispatch 2024-09-21T12:34:50.194 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:49 smithi079 ceph-mon[30023]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi079", "name": "osd_memory_target"}]: dispatch 2024-09-21T12:34:50.194 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:49 smithi079 ceph-mon[30023]: from='mgr.15351 172.21.15.79:0/1857156883' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:34:50.194 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:49 smithi079 ceph-mon[30023]: from='mgr.15351 172.21.15.79:0/1857156883' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:34:50.194 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:49 smithi079 ceph-mon[30023]: Updating smithi079:/etc/ceph/ceph.conf 2024-09-21T12:34:50.194 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:49 smithi079 ceph-mon[30023]: Updating smithi104:/etc/ceph/ceph.conf 2024-09-21T12:34:50.194 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:49 smithi079 ceph-mon[30023]: from='mgr.15351 ' entity='mgr.y' 2024-09-21T12:34:50.194 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:49 smithi079 ceph-mon[30023]: from='mgr.15351 ' entity='mgr.y' 2024-09-21T12:34:50.194 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:49 smithi079 ceph-mon[30023]: Updating smithi079:/var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T12:34:50.194 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:49 smithi079 ceph-mon[30023]: from='mgr.15351 ' entity='mgr.y' 2024-09-21T12:34:50.194 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:49 smithi079 ceph-mon[30023]: Updating smithi104:/var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T12:34:50.194 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:49 smithi079 ceph-mon[30023]: from='mgr.15351 ' entity='mgr.y' 2024-09-21T12:34:50.195 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:49 smithi079 ceph-mon[30023]: from='mgr.15351 ' entity='mgr.y' 2024-09-21T12:34:50.195 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:49 smithi079 ceph-mon[30023]: from='mgr.15351 ' entity='mgr.y' 2024-09-21T12:34:50.195 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:49 smithi079 ceph-mon[30023]: from='mgr.15351 ' entity='mgr.y' 2024-09-21T12:34:50.195 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:49 smithi079 ceph-mon[30023]: from='mgr.15351 ' entity='mgr.y' 2024-09-21T12:34:50.195 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:49 smithi079 ceph-mon[30023]: from='mgr.15351 ' entity='mgr.y' 2024-09-21T12:34:50.195 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:49 smithi079 ceph-mon[30023]: from='mgr.15351 ' entity='mgr.y' 2024-09-21T12:34:50.195 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:49 smithi079 ceph-mon[30023]: from='mgr.15351 ' entity='mgr.y' 2024-09-21T12:34:50.195 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:49 smithi079 ceph-mon[30023]: from='mgr.15351 ' entity='mgr.y' 2024-09-21T12:34:50.195 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:49 smithi079 ceph-mon[23699]: [21/Sep/2024:12:34:48] ENGINE Serving on https://172.21.15.79:7150 2024-09-21T12:34:50.196 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:49 smithi079 ceph-mon[23699]: [21/Sep/2024:12:34:48] ENGINE Bus STARTED 2024-09-21T12:34:50.197 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:49 smithi079 ceph-mon[23699]: [21/Sep/2024:12:34:48] ENGINE Error in HTTPServer.serve 2024-09-21T12:34:50.197 INFO:journalctl@ceph.mon.a.smithi079.stdout: Traceback (most recent call last): 2024-09-21T12:34:50.197 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T12:34:50.197 INFO:journalctl@ceph.mon.a.smithi079.stdout: self._connections.run(self.expiration_interval) 2024-09-21T12:34:50.197 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T12:34:50.197 INFO:journalctl@ceph.mon.a.smithi079.stdout: self._run(expiration_interval) 2024-09-21T12:34:50.197 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T12:34:50.197 INFO:journalctl@ceph.mon.a.smithi079.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T12:34:50.197 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T12:34:50.197 INFO:journalctl@ceph.mon.a.smithi079.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T12:34:50.197 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T12:34:50.198 INFO:journalctl@ceph.mon.a.smithi079.stdout: s = self.context.wrap_socket( 2024-09-21T12:34:50.198 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T12:34:50.198 INFO:journalctl@ceph.mon.a.smithi079.stdout: return self.sslsocket_class._create( 2024-09-21T12:34:50.198 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T12:34:50.198 INFO:journalctl@ceph.mon.a.smithi079.stdout: self.do_handshake() 2024-09-21T12:34:50.198 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T12:34:50.198 INFO:journalctl@ceph.mon.a.smithi079.stdout: self._sslobj.do_handshake() 2024-09-21T12:34:50.198 INFO:journalctl@ceph.mon.a.smithi079.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T12:34:50.198 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:49 smithi079 ceph-mon[23699]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 3 failed cephadm daemon(s)) 2024-09-21T12:34:50.198 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:49 smithi079 ceph-mon[23699]: Cluster is now healthy 2024-09-21T12:34:50.198 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:49 smithi079 ceph-mon[23699]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:34:50.198 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:49 smithi079 ceph-mon[23699]: from='mgr.15351 172.21.15.79:0/1857156883' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi079", "name": "osd_memory_target"}]: dispatch 2024-09-21T12:34:50.198 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:49 smithi079 ceph-mon[23699]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi079", "name": "osd_memory_target"}]: dispatch 2024-09-21T12:34:50.198 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:49 smithi079 ceph-mon[23699]: from='mgr.15351 172.21.15.79:0/1857156883' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:34:50.199 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:49 smithi079 ceph-mon[23699]: from='mgr.15351 172.21.15.79:0/1857156883' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:34:50.199 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:49 smithi079 ceph-mon[23699]: Updating smithi079:/etc/ceph/ceph.conf 2024-09-21T12:34:50.199 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:49 smithi079 ceph-mon[23699]: Updating smithi104:/etc/ceph/ceph.conf 2024-09-21T12:34:50.199 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:49 smithi079 ceph-mon[23699]: from='mgr.15351 ' entity='mgr.y' 2024-09-21T12:34:50.199 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:49 smithi079 ceph-mon[23699]: from='mgr.15351 ' entity='mgr.y' 2024-09-21T12:34:50.199 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:49 smithi079 ceph-mon[23699]: Updating smithi079:/var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T12:34:50.199 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:49 smithi079 ceph-mon[23699]: from='mgr.15351 ' entity='mgr.y' 2024-09-21T12:34:50.199 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:49 smithi079 ceph-mon[23699]: Updating smithi104:/var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T12:34:50.199 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:49 smithi079 ceph-mon[23699]: from='mgr.15351 ' entity='mgr.y' 2024-09-21T12:34:50.199 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:49 smithi079 ceph-mon[23699]: from='mgr.15351 ' entity='mgr.y' 2024-09-21T12:34:50.199 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:49 smithi079 ceph-mon[23699]: from='mgr.15351 ' entity='mgr.y' 2024-09-21T12:34:50.199 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:49 smithi079 ceph-mon[23699]: from='mgr.15351 ' entity='mgr.y' 2024-09-21T12:34:50.199 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:49 smithi079 ceph-mon[23699]: from='mgr.15351 ' entity='mgr.y' 2024-09-21T12:34:50.199 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:49 smithi079 ceph-mon[23699]: from='mgr.15351 ' entity='mgr.y' 2024-09-21T12:34:50.200 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:49 smithi079 ceph-mon[23699]: from='mgr.15351 ' entity='mgr.y' 2024-09-21T12:34:50.200 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:49 smithi079 ceph-mon[23699]: from='mgr.15351 ' entity='mgr.y' 2024-09-21T12:34:50.200 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:49 smithi079 ceph-mon[23699]: from='mgr.15351 ' entity='mgr.y' 2024-09-21T12:34:51.107 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:50 smithi104 ceph-mon[25639]: Updating smithi079:/etc/ceph/ceph.client.admin.keyring 2024-09-21T12:34:51.107 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:50 smithi104 ceph-mon[25639]: Updating smithi104:/etc/ceph/ceph.client.admin.keyring 2024-09-21T12:34:51.107 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:50 smithi104 ceph-mon[25639]: Updating smithi079:/var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T12:34:51.107 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:50 smithi104 ceph-mon[25639]: Updating smithi104:/var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T12:34:51.107 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:50 smithi104 ceph-mon[25639]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:34:51.107 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:50 smithi104 ceph-mon[25639]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-21T12:34:51.107 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:50 smithi104 ceph-mon[25639]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-21T12:34:51.107 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:50 smithi104 ceph-mon[25639]: Health check failed: 3 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-09-21T12:34:51.107 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:50 smithi104 ceph-mon[25639]: Reconfiguring daemon prometheus.a on smithi104 2024-09-21T12:34:51.107 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:50 smithi104 ceph-mon[25639]: mgrmap e30: y(active, since 3s) 2024-09-21T12:34:51.124 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:50 smithi079 ceph-mon[23699]: Updating smithi079:/etc/ceph/ceph.client.admin.keyring 2024-09-21T12:34:51.125 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:50 smithi079 ceph-mon[23699]: Updating smithi104:/etc/ceph/ceph.client.admin.keyring 2024-09-21T12:34:51.125 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:50 smithi079 ceph-mon[23699]: Updating smithi079:/var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T12:34:51.125 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:50 smithi079 ceph-mon[23699]: Updating smithi104:/var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T12:34:51.125 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:50 smithi079 ceph-mon[23699]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:34:51.125 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:50 smithi079 ceph-mon[23699]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-21T12:34:51.125 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:50 smithi079 ceph-mon[23699]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-21T12:34:51.125 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:50 smithi079 ceph-mon[23699]: Health check failed: 3 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-09-21T12:34:51.125 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:50 smithi079 ceph-mon[23699]: Reconfiguring daemon prometheus.a on smithi104 2024-09-21T12:34:51.125 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:50 smithi079 ceph-mon[23699]: mgrmap e30: y(active, since 3s) 2024-09-21T12:34:51.125 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:50 smithi079 ceph-mon[30023]: Updating smithi079:/etc/ceph/ceph.client.admin.keyring 2024-09-21T12:34:51.125 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:50 smithi079 ceph-mon[30023]: Updating smithi104:/etc/ceph/ceph.client.admin.keyring 2024-09-21T12:34:51.125 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:50 smithi079 ceph-mon[30023]: Updating smithi079:/var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T12:34:51.126 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:50 smithi079 ceph-mon[30023]: Updating smithi104:/var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T12:34:51.126 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:50 smithi079 ceph-mon[30023]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:34:51.126 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:50 smithi079 ceph-mon[30023]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-21T12:34:51.126 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:50 smithi079 ceph-mon[30023]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-21T12:34:51.126 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:50 smithi079 ceph-mon[30023]: Health check failed: 3 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-09-21T12:34:51.126 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:50 smithi079 ceph-mon[30023]: Reconfiguring daemon prometheus.a on smithi104 2024-09-21T12:34:51.126 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:50 smithi079 ceph-mon[30023]: mgrmap e30: y(active, since 3s) 2024-09-21T12:34:52.378 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:52 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: [21/Sep/2024:12:34:52] ENGINE Bus STARTING 2024-09-21T12:34:52.378 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:52 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: CherryPy Checker: 2024-09-21T12:34:52.378 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:52 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: The Application mounted at '' has an empty config. 2024-09-21T12:34:52.378 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:52 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: 2024-09-21T12:34:52.378 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:52 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: [21/Sep/2024:12:34:52] ENGINE Serving on http://:::9283 2024-09-21T12:34:52.676 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:52 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: [21/Sep/2024:12:34:52] ENGINE Bus STARTED 2024-09-21T12:34:53.156 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:52 smithi079 ceph-mon[23699]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:34:53.156 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:52 smithi079 ceph-mon[23699]: Standby manager daemon x started 2024-09-21T12:34:53.156 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:52 smithi079 ceph-mon[23699]: from='mgr.? 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-21T12:34:53.156 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:52 smithi079 ceph-mon[23699]: from='mgr.? 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-21T12:34:53.156 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:52 smithi079 ceph-mon[23699]: from='mgr.? 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-21T12:34:53.156 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:52 smithi079 ceph-mon[23699]: from='mgr.? 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-21T12:34:53.156 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:52 smithi079 ceph-mon[23699]: from='mgr.15351 ' entity='mgr.y' 2024-09-21T12:34:53.157 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:52 smithi079 ceph-mon[23699]: from='mgr.15351 ' entity='mgr.y' 2024-09-21T12:34:53.157 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:52 smithi079 ceph-mon[23699]: from='mgr.15351 ' entity='mgr.y' 2024-09-21T12:34:53.157 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:52 smithi079 ceph-mon[30023]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:34:53.157 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:52 smithi079 ceph-mon[30023]: Standby manager daemon x started 2024-09-21T12:34:53.157 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:52 smithi079 ceph-mon[30023]: from='mgr.? 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-21T12:34:53.157 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:52 smithi079 ceph-mon[30023]: from='mgr.? 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-21T12:34:53.157 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:52 smithi079 ceph-mon[30023]: from='mgr.? 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-21T12:34:53.157 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:52 smithi079 ceph-mon[30023]: from='mgr.? 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-21T12:34:53.157 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:52 smithi079 ceph-mon[30023]: from='mgr.15351 ' entity='mgr.y' 2024-09-21T12:34:53.157 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:52 smithi079 ceph-mon[30023]: from='mgr.15351 ' entity='mgr.y' 2024-09-21T12:34:53.157 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:52 smithi079 ceph-mon[30023]: from='mgr.15351 ' entity='mgr.y' 2024-09-21T12:34:53.176 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:52 smithi104 ceph-mon[25639]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:34:53.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:52 smithi104 ceph-mon[25639]: Standby manager daemon x started 2024-09-21T12:34:53.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:52 smithi104 ceph-mon[25639]: from='mgr.? 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-21T12:34:53.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:52 smithi104 ceph-mon[25639]: from='mgr.? 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-21T12:34:53.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:52 smithi104 ceph-mon[25639]: from='mgr.? 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-21T12:34:53.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:52 smithi104 ceph-mon[25639]: from='mgr.? 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-21T12:34:53.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:52 smithi104 ceph-mon[25639]: from='mgr.15351 ' entity='mgr.y' 2024-09-21T12:34:53.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:52 smithi104 ceph-mon[25639]: from='mgr.15351 ' entity='mgr.y' 2024-09-21T12:34:53.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:52 smithi104 ceph-mon[25639]: from='mgr.15351 ' entity='mgr.y' 2024-09-21T12:34:54.089 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:53 smithi104 ceph-mon[25639]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:34:54.089 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:53 smithi104 ceph-mon[25639]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:34:54.089 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:53 smithi104 ceph-mon[25639]: mgrmap e31: y(active, since 6s), standbys: x 2024-09-21T12:34:54.089 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:53 smithi104 ceph-mon[25639]: from='mgr.15351 172.21.15.79:0/1857156883' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-21T12:34:54.162 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:53 smithi079 ceph-mon[23699]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:34:54.162 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:53 smithi079 ceph-mon[23699]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:34:54.162 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:53 smithi079 ceph-mon[23699]: mgrmap e31: y(active, since 6s), standbys: x 2024-09-21T12:34:54.162 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:53 smithi079 ceph-mon[23699]: from='mgr.15351 172.21.15.79:0/1857156883' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-21T12:34:54.163 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:53 smithi079 ceph-mon[30023]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:34:54.163 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:53 smithi079 ceph-mon[30023]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:34:54.163 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:53 smithi079 ceph-mon[30023]: mgrmap e31: y(active, since 6s), standbys: x 2024-09-21T12:34:54.163 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:53 smithi079 ceph-mon[30023]: from='mgr.15351 172.21.15.79:0/1857156883' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-21T12:34:56.093 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:55 smithi104 ceph-mon[25639]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-21T12:34:56.146 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:55 smithi079 ceph-mon[23699]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-21T12:34:56.146 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:55 smithi079 ceph-mon[30023]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-21T12:34:57.091 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:34:56 smithi104 systemd[1]: Stopping Ceph prometheus.a for 33168022-7814-11ef-baf6-efdc52797490... 2024-09-21T12:34:57.387 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:57 smithi104 ceph-mon[25639]: from='mgr.15351 ' entity='mgr.y' 2024-09-21T12:34:57.387 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:57 smithi104 ceph-mon[25639]: from='mgr.15351 ' entity='mgr.y' 2024-09-21T12:34:57.387 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:57 smithi104 ceph-mon[25639]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-21T12:34:57.387 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:57 smithi104 ceph-mon[25639]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-21T12:34:57.387 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:57 smithi104 ceph-mon[25639]: from='mgr.15351 ' entity='mgr.y' 2024-09-21T12:34:57.387 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:57 smithi104 ceph-mon[25639]: from='mgr.15351 ' entity='mgr.y' 2024-09-21T12:34:57.387 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:57 smithi104 ceph-mon[25639]: from='mgr.15351 ' entity='mgr.y' 2024-09-21T12:34:57.387 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:57 smithi104 ceph-mon[25639]: from='mgr.15351 ' entity='mgr.y' 2024-09-21T12:34:57.388 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:34:57 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[59471]: ts=2024-09-21T12:34:57.239Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-09-21T12:34:57.388 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:34:57 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[59471]: ts=2024-09-21T12:34:57.239Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-09-21T12:34:57.388 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:34:57 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[59471]: ts=2024-09-21T12:34:57.239Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-09-21T12:34:57.388 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:34:57 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[59471]: ts=2024-09-21T12:34:57.239Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-09-21T12:34:57.388 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:34:57 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[59471]: ts=2024-09-21T12:34:57.239Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-09-21T12:34:57.388 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:34:57 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[59471]: ts=2024-09-21T12:34:57.239Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-09-21T12:34:57.388 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:34:57 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[59471]: ts=2024-09-21T12:34:57.239Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-09-21T12:34:57.388 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:34:57 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[59471]: ts=2024-09-21T12:34:57.239Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-09-21T12:34:57.388 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:34:57 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[59471]: ts=2024-09-21T12:34:57.239Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-09-21T12:34:57.388 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:34:57 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[59471]: ts=2024-09-21T12:34:57.239Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-09-21T12:34:57.389 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:34:57 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[59471]: ts=2024-09-21T12:34:57.239Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-09-21T12:34:57.389 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:34:57 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[59471]: ts=2024-09-21T12:34:57.239Z caller=main.go:1147 level=info msg="See you next time!" 2024-09-21T12:34:57.389 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:34:57 smithi104 podman[61346]: 2024-09-21 12:34:57.249421546 +0000 UTC m=+0.151747423 container died bc3d876e1babfdb2598661dd161e5f63a4c06d3b789495ac6b920f4639321e1f (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T12:34:57.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:57 smithi079 ceph-mon[23699]: from='mgr.15351 ' entity='mgr.y' 2024-09-21T12:34:57.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:57 smithi079 ceph-mon[23699]: from='mgr.15351 ' entity='mgr.y' 2024-09-21T12:34:57.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:57 smithi079 ceph-mon[23699]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-21T12:34:57.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:57 smithi079 ceph-mon[23699]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-21T12:34:57.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:57 smithi079 ceph-mon[23699]: from='mgr.15351 ' entity='mgr.y' 2024-09-21T12:34:57.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:57 smithi079 ceph-mon[23699]: from='mgr.15351 ' entity='mgr.y' 2024-09-21T12:34:57.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:57 smithi079 ceph-mon[23699]: from='mgr.15351 ' entity='mgr.y' 2024-09-21T12:34:57.442 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:57 smithi079 ceph-mon[23699]: from='mgr.15351 ' entity='mgr.y' 2024-09-21T12:34:57.442 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:57 smithi079 ceph-mon[30023]: from='mgr.15351 ' entity='mgr.y' 2024-09-21T12:34:57.442 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:57 smithi079 ceph-mon[30023]: from='mgr.15351 ' entity='mgr.y' 2024-09-21T12:34:57.443 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:57 smithi079 ceph-mon[30023]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 0 B/s wr, 9 op/s 2024-09-21T12:34:57.443 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:57 smithi079 ceph-mon[30023]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-21T12:34:57.443 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:57 smithi079 ceph-mon[30023]: from='mgr.15351 ' entity='mgr.y' 2024-09-21T12:34:57.443 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:57 smithi079 ceph-mon[30023]: from='mgr.15351 ' entity='mgr.y' 2024-09-21T12:34:57.443 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:57 smithi079 ceph-mon[30023]: from='mgr.15351 ' entity='mgr.y' 2024-09-21T12:34:57.443 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:57 smithi079 ceph-mon[30023]: from='mgr.15351 ' entity='mgr.y' 2024-09-21T12:34:57.927 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:34:57 smithi104 podman[61346]: 2024-09-21 12:34:57.643500006 +0000 UTC m=+0.545825870 container cleanup bc3d876e1babfdb2598661dd161e5f63a4c06d3b789495ac6b920f4639321e1f (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T12:34:57.927 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:34:57 smithi104 bash[61346]: ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a 2024-09-21T12:34:57.927 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:34:57 smithi104 podman[61410]: 2024-09-21 12:34:57.869741232 +0000 UTC m=+0.622423447 container remove bc3d876e1babfdb2598661dd161e5f63a4c06d3b789495ac6b920f4639321e1f (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T12:34:58.308 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:58 smithi079 ceph-mon[23699]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 32 KiB/s rd, 0 B/s wr, 13 op/s 2024-09-21T12:34:58.309 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:58 smithi079 ceph-mon[23699]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 16 op/s 2024-09-21T12:34:58.309 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:58 smithi079 ceph-mon[23699]: Health check update: 2 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-09-21T12:34:58.310 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:58 smithi079 ceph-mon[23699]: from='mgr.15351 ' entity='mgr.y' 2024-09-21T12:34:58.310 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:58 smithi079 ceph-mon[23699]: from='mgr.15351 ' entity='mgr.y' 2024-09-21T12:34:58.310 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:58 smithi079 ceph-mon[23699]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 39 KiB/s rd, 0 B/s wr, 15 op/s 2024-09-21T12:34:58.310 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:58 smithi079 ceph-mon[23699]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:34:58.310 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:58 smithi079 ceph-mon[23699]: from='mgr.15351 ' entity='mgr.y' 2024-09-21T12:34:58.310 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:58 smithi079 ceph-mon[30023]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 32 KiB/s rd, 0 B/s wr, 13 op/s 2024-09-21T12:34:58.310 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:58 smithi079 ceph-mon[30023]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 16 op/s 2024-09-21T12:34:58.310 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:58 smithi079 ceph-mon[30023]: Health check update: 2 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-09-21T12:34:58.310 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:58 smithi079 ceph-mon[30023]: from='mgr.15351 ' entity='mgr.y' 2024-09-21T12:34:58.310 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:58 smithi079 ceph-mon[30023]: from='mgr.15351 ' entity='mgr.y' 2024-09-21T12:34:58.310 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:58 smithi079 ceph-mon[30023]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 39 KiB/s rd, 0 B/s wr, 15 op/s 2024-09-21T12:34:58.311 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:58 smithi079 ceph-mon[30023]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:34:58.311 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:58 smithi079 ceph-mon[30023]: from='mgr.15351 ' entity='mgr.y' 2024-09-21T12:34:58.311 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:34:58 smithi104 systemd[1]: ceph-33168022-7814-11ef-baf6-efdc52797490@prometheus.a.service: Deactivated successfully. 2024-09-21T12:34:58.311 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:34:58 smithi104 systemd[1]: Stopped Ceph prometheus.a for 33168022-7814-11ef-baf6-efdc52797490. 2024-09-21T12:34:58.311 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:34:58 smithi104 systemd[1]: ceph-33168022-7814-11ef-baf6-efdc52797490@prometheus.a.service: Consumed 1.047s CPU time. 2024-09-21T12:34:58.311 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:34:58 smithi104 systemd[1]: Starting Ceph prometheus.a for 33168022-7814-11ef-baf6-efdc52797490... 2024-09-21T12:34:58.311 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:34:58 smithi104 podman[61535]: 2024-09-21 12:34:58.228935553 +0000 UTC m=+0.104082823 container create 724fc2fdc4783d91e1bcb13638f01f99aa4a02d728af22c7120b25bab269f5f9 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T12:34:58.312 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:34:58 smithi104 podman[61535]: 2024-09-21 12:34:58.14165312 +0000 UTC m=+0.016800406 image pull a07b618ecd1dce142bce4c52f0e80982eaf1f14265a2415c2d35978ccaa0a464 quay.io/prometheus/prometheus:v2.43.0 2024-09-21T12:34:58.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:58 smithi104 ceph-mon[25639]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 32 KiB/s rd, 0 B/s wr, 13 op/s 2024-09-21T12:34:58.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:58 smithi104 ceph-mon[25639]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 16 op/s 2024-09-21T12:34:58.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:58 smithi104 ceph-mon[25639]: Health check update: 2 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-09-21T12:34:58.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:58 smithi104 ceph-mon[25639]: from='mgr.15351 ' entity='mgr.y' 2024-09-21T12:34:58.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:58 smithi104 ceph-mon[25639]: from='mgr.15351 ' entity='mgr.y' 2024-09-21T12:34:58.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:58 smithi104 ceph-mon[25639]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail; 39 KiB/s rd, 0 B/s wr, 15 op/s 2024-09-21T12:34:58.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:58 smithi104 ceph-mon[25639]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:34:58.312 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:58 smithi104 ceph-mon[25639]: from='mgr.15351 ' entity='mgr.y' 2024-09-21T12:34:58.673 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:34:58 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[70883]: 2024-09-21T12:34:58.605+0000 7f834513c640 -1 mgr handle_mgr_map I was active but no longer am 2024-09-21T12:34:58.673 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:34:58 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[70883]: ignoring --setuser ceph since I am not root 2024-09-21T12:34:58.674 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:34:58 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[70883]: ignoring --setgroup ceph since I am not root 2024-09-21T12:34:58.676 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:58 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: [21/Sep/2024:12:34:58] ENGINE Bus STOPPING 2024-09-21T12:34:58.677 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:34:58 smithi104 podman[61535]: 2024-09-21 12:34:58.422608366 +0000 UTC m=+0.297755633 container init 724fc2fdc4783d91e1bcb13638f01f99aa4a02d728af22c7120b25bab269f5f9 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T12:34:58.677 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:34:58 smithi104 podman[61535]: 2024-09-21 12:34:58.425478883 +0000 UTC m=+0.300626151 container start 724fc2fdc4783d91e1bcb13638f01f99aa4a02d728af22c7120b25bab269f5f9 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T12:34:58.677 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:34:58 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[61550]: ts=2024-09-21T12:34:58.448Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-09-21T12:34:58.677 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:34:58 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[61550]: ts=2024-09-21T12:34:58.448Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-09-21T12:34:58.677 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:34:58 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[61550]: ts=2024-09-21T12:34:58.448Z caller=main.go:570 level=info host_details="(Linux 5.14.0-508.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024 x86_64 smithi104 (none))" 2024-09-21T12:34:58.677 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:34:58 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[61550]: ts=2024-09-21T12:34:58.449Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-09-21T12:34:58.677 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:34:58 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[61550]: ts=2024-09-21T12:34:58.449Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-09-21T12:34:58.677 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:34:58 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[61550]: ts=2024-09-21T12:34:58.450Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-09-21T12:34:58.677 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:34:58 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[61550]: ts=2024-09-21T12:34:58.451Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-09-21T12:34:58.677 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:34:58 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[61550]: ts=2024-09-21T12:34:58.452Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-09-21T12:34:58.677 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:34:58 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[61550]: ts=2024-09-21T12:34:58.452Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-09-21T12:34:58.678 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:34:58 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[61550]: ts=2024-09-21T12:34:58.454Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-09-21T12:34:58.678 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:34:58 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[61550]: ts=2024-09-21T12:34:58.454Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.463µs 2024-09-21T12:34:58.678 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:34:58 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[61550]: ts=2024-09-21T12:34:58.454Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-09-21T12:34:58.678 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:34:58 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[61550]: ts=2024-09-21T12:34:58.455Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=3 2024-09-21T12:34:58.678 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:34:58 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[61550]: ts=2024-09-21T12:34:58.464Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=3 2024-09-21T12:34:58.678 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:34:58 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[61550]: ts=2024-09-21T12:34:58.465Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=2 maxSegment=3 2024-09-21T12:34:58.678 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:34:58 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[61550]: ts=2024-09-21T12:34:58.465Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=3 maxSegment=3 2024-09-21T12:34:58.678 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:34:58 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[61550]: ts=2024-09-21T12:34:58.465Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=19.738µs wal_replay_duration=10.779897ms wbl_replay_duration=100ns total_replay_duration=10.816591ms 2024-09-21T12:34:58.678 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:34:58 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[61550]: ts=2024-09-21T12:34:58.466Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-09-21T12:34:58.678 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:34:58 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[61550]: ts=2024-09-21T12:34:58.466Z caller=main.go:1029 level=info msg="TSDB started" 2024-09-21T12:34:58.678 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:34:58 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[61550]: ts=2024-09-21T12:34:58.466Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-09-21T12:34:58.678 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:34:58 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[61550]: ts=2024-09-21T12:34:58.487Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=20.404143ms db_storage=769ns remote_storage=1.435µs web_handler=431ns query_engine=540ns scrape=386.879µs scrape_sd=145.214µs notify=17.996µs notify_sd=24.457µs rules=19.461309ms tracing=6.467µs 2024-09-21T12:34:58.678 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:34:58 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[61550]: ts=2024-09-21T12:34:58.487Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-09-21T12:34:58.679 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:34:58 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[61550]: ts=2024-09-21T12:34:58.487Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-09-21T12:34:58.679 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:34:58 smithi104 bash[61535]: 724fc2fdc4783d91e1bcb13638f01f99aa4a02d728af22c7120b25bab269f5f9 2024-09-21T12:34:58.679 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:34:58 smithi104 systemd[1]: Started Ceph prometheus.a for 33168022-7814-11ef-baf6-efdc52797490. 2024-09-21T12:34:58.941 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:34:58 smithi079 ceph-mgr[70887]: -- 172.21.15.79:0/713549684 <== mon.0 v2:172.21.15.79:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55bd5e8b70e0 con 0x55bd5e895400 2024-09-21T12:34:58.985 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:58 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: [21/Sep/2024:12:34:58] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-21T12:34:58.985 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:58 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: [21/Sep/2024:12:34:58] ENGINE Bus STOPPED 2024-09-21T12:34:59.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:59 smithi104 ceph-mon[25639]: from='mgr.15351 ' entity='mgr.y' 2024-09-21T12:34:59.678 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:59 smithi104 ceph-mon[25639]: from='mgr.15351 ' entity='mgr.y' 2024-09-21T12:34:59.678 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:59 smithi104 ceph-mon[25639]: from='mgr.15351 172.21.15.79:0/1857156883' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-21T12:34:59.678 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:59 smithi104 ceph-mon[25639]: from='mgr.15351 172.21.15.79:0/1857156883' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:34:59.678 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:59 smithi104 ceph-mon[25639]: from='mgr.15351 172.21.15.79:0/1857156883' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-09-21T12:34:59.678 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:59 smithi104 ceph-mon[25639]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-09-21T12:34:59.678 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:59 smithi104 ceph-mon[25639]: Activating manager daemon x 2024-09-21T12:34:59.678 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:59 smithi104 ceph-mon[25639]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T12:34:59.678 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:59 smithi104 ceph-mon[25639]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T12:34:59.678 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:59 smithi104 ceph-mon[25639]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T12:34:59.678 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:59 smithi104 ceph-mon[25639]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-21T12:34:59.678 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:59 smithi104 ceph-mon[25639]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T12:34:59.678 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:59 smithi104 ceph-mon[25639]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T12:34:59.679 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:59 smithi104 ceph-mon[25639]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T12:34:59.679 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:59 smithi104 ceph-mon[25639]: osdmap e105: 8 total, 8 up, 8 in 2024-09-21T12:34:59.679 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:59 smithi104 ceph-mon[25639]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T12:34:59.679 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:59 smithi104 ceph-mon[25639]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T12:34:59.679 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:59 smithi104 ceph-mon[25639]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T12:34:59.679 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:59 smithi104 ceph-mon[25639]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T12:34:59.679 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:59 smithi104 ceph-mon[25639]: from='mgr.15351 ' entity='mgr.y' cmd='[{"prefix": "mgr fail", "who": "y"}]': finished 2024-09-21T12:34:59.679 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:59 smithi104 ceph-mon[25639]: mgrmap e32: x(active, starting, since 0.01412s) 2024-09-21T12:34:59.679 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:59 smithi104 ceph-mon[25639]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T12:34:59.679 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:59 smithi104 ceph-mon[25639]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-21T12:34:59.679 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:59 smithi104 ceph-mon[25639]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-21T12:34:59.679 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:59 smithi104 ceph-mon[25639]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-21T12:34:59.679 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:59 smithi104 ceph-mon[25639]: Manager daemon x is now available 2024-09-21T12:34:59.679 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:59 smithi104 ceph-mon[25639]: from='client.? 172.21.15.79:0/3450501810' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-21T12:34:59.679 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:59 smithi104 ceph-mon[25639]: from='mgr.25054 ' entity='mgr.x' 2024-09-21T12:34:59.680 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:59 smithi104 ceph-mon[25639]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:34:59.680 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:59 smithi104 ceph-mon[25639]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi104", "caps": []}]: dispatch 2024-09-21T12:34:59.680 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:59 smithi104 ceph-mon[25639]: from='mgr.25054 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi104", "caps": []}]: dispatch 2024-09-21T12:34:59.680 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:59 smithi104 ceph-mon[25639]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi079", "caps": []}]: dispatch 2024-09-21T12:34:59.680 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:59 smithi104 ceph-mon[25639]: from='mgr.25054 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi079", "caps": []}]: dispatch 2024-09-21T12:34:59.680 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:59 smithi104 ceph-mon[25639]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T12:34:59.680 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:59 smithi104 ceph-mon[25639]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T12:34:59.680 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:59 smithi104 ceph-mon[25639]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-09-21T12:34:59.680 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:59 smithi104 ceph-mon[25639]: from='mgr.25054 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-09-21T12:34:59.680 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:59 smithi104 ceph-mon[25639]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-09-21T12:34:59.680 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:59 smithi104 ceph-mon[25639]: from='mgr.25054 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-09-21T12:34:59.680 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:59 smithi104 ceph-mon[25639]: from='client.? 172.21.15.79:0/1931925635' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:6801/1347188378"}]: dispatch 2024-09-21T12:34:59.680 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:34:59 smithi104 ceph-mon[25639]: from='client.? ' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:6801/1347188378"}]: dispatch 2024-09-21T12:34:59.692 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:59 smithi079 ceph-mon[30023]: from='mgr.15351 ' entity='mgr.y' 2024-09-21T12:34:59.692 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:59 smithi079 ceph-mon[30023]: from='mgr.15351 ' entity='mgr.y' 2024-09-21T12:34:59.692 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:59 smithi079 ceph-mon[30023]: from='mgr.15351 172.21.15.79:0/1857156883' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-21T12:34:59.692 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:59 smithi079 ceph-mon[30023]: from='mgr.15351 172.21.15.79:0/1857156883' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:34:59.692 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:59 smithi079 ceph-mon[30023]: from='mgr.15351 172.21.15.79:0/1857156883' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-09-21T12:34:59.692 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:59 smithi079 ceph-mon[30023]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-09-21T12:34:59.692 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:59 smithi079 ceph-mon[30023]: Activating manager daemon x 2024-09-21T12:34:59.693 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:59 smithi079 ceph-mon[30023]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T12:34:59.693 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:59 smithi079 ceph-mon[30023]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T12:34:59.693 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:59 smithi079 ceph-mon[30023]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T12:34:59.693 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:59 smithi079 ceph-mon[30023]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-21T12:34:59.693 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:59 smithi079 ceph-mon[30023]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T12:34:59.693 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:59 smithi079 ceph-mon[30023]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T12:34:59.693 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:59 smithi079 ceph-mon[30023]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T12:34:59.693 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:59 smithi079 ceph-mon[30023]: osdmap e105: 8 total, 8 up, 8 in 2024-09-21T12:34:59.693 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:59 smithi079 ceph-mon[30023]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T12:34:59.693 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:59 smithi079 ceph-mon[30023]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T12:34:59.693 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:59 smithi079 ceph-mon[30023]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T12:34:59.693 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:59 smithi079 ceph-mon[30023]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T12:34:59.693 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:59 smithi079 ceph-mon[30023]: from='mgr.15351 ' entity='mgr.y' cmd='[{"prefix": "mgr fail", "who": "y"}]': finished 2024-09-21T12:34:59.693 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:59 smithi079 ceph-mon[30023]: mgrmap e32: x(active, starting, since 0.01412s) 2024-09-21T12:34:59.693 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:59 smithi079 ceph-mon[30023]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T12:34:59.694 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:59 smithi079 ceph-mon[30023]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-21T12:34:59.694 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:59 smithi079 ceph-mon[30023]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-21T12:34:59.694 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:59 smithi079 ceph-mon[30023]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-21T12:34:59.694 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:59 smithi079 ceph-mon[30023]: Manager daemon x is now available 2024-09-21T12:34:59.694 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:59 smithi079 ceph-mon[30023]: from='client.? 172.21.15.79:0/3450501810' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-21T12:34:59.694 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:59 smithi079 ceph-mon[30023]: from='mgr.25054 ' entity='mgr.x' 2024-09-21T12:34:59.694 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:59 smithi079 ceph-mon[30023]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:34:59.694 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:59 smithi079 ceph-mon[23699]: from='mgr.15351 ' entity='mgr.y' 2024-09-21T12:34:59.694 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:59 smithi079 ceph-mon[23699]: from='mgr.15351 ' entity='mgr.y' 2024-09-21T12:34:59.694 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:59 smithi079 ceph-mon[23699]: from='mgr.15351 172.21.15.79:0/1857156883' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-21T12:34:59.694 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:59 smithi079 ceph-mon[23699]: from='mgr.15351 172.21.15.79:0/1857156883' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:34:59.695 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:59 smithi079 ceph-mon[23699]: from='mgr.15351 172.21.15.79:0/1857156883' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-09-21T12:34:59.695 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:59 smithi079 ceph-mon[23699]: from='mgr.15351 ' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-09-21T12:34:59.695 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:59 smithi079 ceph-mon[23699]: Activating manager daemon x 2024-09-21T12:34:59.695 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:59 smithi079 ceph-mon[23699]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T12:34:59.695 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:59 smithi079 ceph-mon[23699]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T12:34:59.695 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:59 smithi079 ceph-mon[23699]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T12:34:59.695 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:59 smithi079 ceph-mon[23699]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-21T12:34:59.695 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:59 smithi079 ceph-mon[23699]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T12:34:59.695 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:59 smithi079 ceph-mon[23699]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T12:34:59.695 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:59 smithi079 ceph-mon[23699]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T12:34:59.695 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:59 smithi079 ceph-mon[23699]: osdmap e105: 8 total, 8 up, 8 in 2024-09-21T12:34:59.695 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:59 smithi079 ceph-mon[23699]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T12:34:59.695 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:59 smithi079 ceph-mon[23699]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T12:34:59.695 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:59 smithi079 ceph-mon[23699]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T12:34:59.695 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:59 smithi079 ceph-mon[23699]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T12:34:59.696 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:59 smithi079 ceph-mon[23699]: from='mgr.15351 ' entity='mgr.y' cmd='[{"prefix": "mgr fail", "who": "y"}]': finished 2024-09-21T12:34:59.696 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:59 smithi079 ceph-mon[23699]: mgrmap e32: x(active, starting, since 0.01412s) 2024-09-21T12:34:59.696 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:59 smithi079 ceph-mon[23699]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T12:34:59.696 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:59 smithi079 ceph-mon[23699]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-21T12:34:59.696 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:59 smithi079 ceph-mon[23699]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-21T12:34:59.696 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:59 smithi079 ceph-mon[23699]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-21T12:34:59.696 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:59 smithi079 ceph-mon[23699]: Manager daemon x is now available 2024-09-21T12:34:59.696 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:59 smithi079 ceph-mon[23699]: from='client.? 172.21.15.79:0/3450501810' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-21T12:34:59.696 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:59 smithi079 ceph-mon[23699]: from='mgr.25054 ' entity='mgr.x' 2024-09-21T12:34:59.696 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:59 smithi079 ceph-mon[23699]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:34:59.696 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:59 smithi079 ceph-mon[30023]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi104", "caps": []}]: dispatch 2024-09-21T12:34:59.697 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:59 smithi079 ceph-mon[30023]: from='mgr.25054 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi104", "caps": []}]: dispatch 2024-09-21T12:34:59.697 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:59 smithi079 ceph-mon[30023]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi079", "caps": []}]: dispatch 2024-09-21T12:34:59.697 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:59 smithi079 ceph-mon[30023]: from='mgr.25054 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi079", "caps": []}]: dispatch 2024-09-21T12:34:59.697 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:59 smithi079 ceph-mon[30023]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T12:34:59.697 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:59 smithi079 ceph-mon[30023]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T12:34:59.697 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:59 smithi079 ceph-mon[30023]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-09-21T12:34:59.697 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:59 smithi079 ceph-mon[30023]: from='mgr.25054 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-09-21T12:34:59.697 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:59 smithi079 ceph-mon[30023]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-09-21T12:34:59.697 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:59 smithi079 ceph-mon[30023]: from='mgr.25054 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-09-21T12:34:59.697 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:59 smithi079 ceph-mon[30023]: from='client.? 172.21.15.79:0/1931925635' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:6801/1347188378"}]: dispatch 2024-09-21T12:34:59.697 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:34:59 smithi079 ceph-mon[30023]: from='client.? ' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:6801/1347188378"}]: dispatch 2024-09-21T12:34:59.697 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:59 smithi079 ceph-mon[23699]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi104", "caps": []}]: dispatch 2024-09-21T12:34:59.697 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:59 smithi079 ceph-mon[23699]: from='mgr.25054 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi104", "caps": []}]: dispatch 2024-09-21T12:34:59.697 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:59 smithi079 ceph-mon[23699]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi079", "caps": []}]: dispatch 2024-09-21T12:34:59.698 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:59 smithi079 ceph-mon[23699]: from='mgr.25054 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi079", "caps": []}]: dispatch 2024-09-21T12:34:59.698 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:59 smithi079 ceph-mon[23699]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T12:34:59.698 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:59 smithi079 ceph-mon[23699]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T12:34:59.698 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:59 smithi079 ceph-mon[23699]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-09-21T12:34:59.698 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:59 smithi079 ceph-mon[23699]: from='mgr.25054 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-09-21T12:34:59.698 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:59 smithi079 ceph-mon[23699]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-09-21T12:34:59.698 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:59 smithi079 ceph-mon[23699]: from='mgr.25054 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-09-21T12:34:59.698 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:59 smithi079 ceph-mon[23699]: from='client.? 172.21.15.79:0/1931925635' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:6801/1347188378"}]: dispatch 2024-09-21T12:34:59.698 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:34:59 smithi079 ceph-mon[23699]: from='client.? ' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:6801/1347188378"}]: dispatch 2024-09-21T12:35:00.057 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:59 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: [21/Sep/2024:12:34:59] ENGINE Bus STARTING 2024-09-21T12:35:00.058 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:59 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: [21/Sep/2024:12:34:59] ENGINE Serving on http://:::9283 2024-09-21T12:35:00.058 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:34:59 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: [21/Sep/2024:12:34:59] ENGINE Bus STARTED 2024-09-21T12:35:00.849 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:00 smithi079 ceph-mon[30023]: mgrmap e33: x(active, since 1.01035s) 2024-09-21T12:35:00.849 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:00 smithi079 ceph-mon[30023]: from='mon.? -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T12:35:00.849 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:00 smithi079 ceph-mon[30023]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:35:00.849 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:00 smithi079 ceph-mon[30023]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi079", "name": "osd_memory_target"}]: dispatch 2024-09-21T12:35:00.849 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:00 smithi079 ceph-mon[30023]: from='mgr.25054 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi079", "name": "osd_memory_target"}]: dispatch 2024-09-21T12:35:00.849 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:00 smithi079 ceph-mon[30023]: from='mgr.25054 ' entity='mgr.x' 2024-09-21T12:35:00.849 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:00 smithi079 ceph-mon[30023]: from='client.? ' entity='client.iscsi.foo.smithi079.vlhrek' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:6801/1347188378"}]': finished 2024-09-21T12:35:00.849 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:00 smithi079 ceph-mon[30023]: osdmap e106: 8 total, 8 up, 8 in 2024-09-21T12:35:00.849 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:00 smithi079 ceph-mon[30023]: from='client.? 172.21.15.79:0/3364742718' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:6800/1347188378"}]: dispatch 2024-09-21T12:35:00.849 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:00 smithi079 ceph-mon[23699]: mgrmap e33: x(active, since 1.01035s) 2024-09-21T12:35:00.850 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:00 smithi079 ceph-mon[23699]: from='mon.? -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T12:35:00.850 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:00 smithi079 ceph-mon[23699]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:35:00.850 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:00 smithi079 ceph-mon[23699]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi079", "name": "osd_memory_target"}]: dispatch 2024-09-21T12:35:00.850 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:00 smithi079 ceph-mon[23699]: from='mgr.25054 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi079", "name": "osd_memory_target"}]: dispatch 2024-09-21T12:35:00.850 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:00 smithi079 ceph-mon[23699]: from='mgr.25054 ' entity='mgr.x' 2024-09-21T12:35:00.850 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:00 smithi079 ceph-mon[23699]: from='client.? ' entity='client.iscsi.foo.smithi079.vlhrek' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:6801/1347188378"}]': finished 2024-09-21T12:35:00.850 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:00 smithi079 ceph-mon[23699]: osdmap e106: 8 total, 8 up, 8 in 2024-09-21T12:35:00.850 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:00 smithi079 ceph-mon[23699]: from='client.? 172.21.15.79:0/3364742718' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:6800/1347188378"}]: dispatch 2024-09-21T12:35:00.926 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:00 smithi104 ceph-mon[25639]: mgrmap e33: x(active, since 1.01035s) 2024-09-21T12:35:00.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:00 smithi104 ceph-mon[25639]: from='mon.? -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T12:35:00.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:00 smithi104 ceph-mon[25639]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:35:00.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:00 smithi104 ceph-mon[25639]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi079", "name": "osd_memory_target"}]: dispatch 2024-09-21T12:35:00.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:00 smithi104 ceph-mon[25639]: from='mgr.25054 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi079", "name": "osd_memory_target"}]: dispatch 2024-09-21T12:35:00.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:00 smithi104 ceph-mon[25639]: from='mgr.25054 ' entity='mgr.x' 2024-09-21T12:35:00.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:00 smithi104 ceph-mon[25639]: from='client.? ' entity='client.iscsi.foo.smithi079.vlhrek' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:6801/1347188378"}]': finished 2024-09-21T12:35:00.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:00 smithi104 ceph-mon[25639]: osdmap e106: 8 total, 8 up, 8 in 2024-09-21T12:35:00.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:00 smithi104 ceph-mon[25639]: from='client.? 172.21.15.79:0/3364742718' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:6800/1347188378"}]: dispatch 2024-09-21T12:35:00.927 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:00 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: 2024-09-21T12:35:00.444+0000 7ff04e41b640 -1 log_channel(cephadm) log [ERR] : [21/Sep/2024:12:35:00] ENGINE Error in HTTPServer.serve 2024-09-21T12:35:00.928 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:00 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: Traceback (most recent call last): 2024-09-21T12:35:00.928 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:00 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T12:35:00.928 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:00 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: self._connections.run(self.expiration_interval) 2024-09-21T12:35:00.928 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:00 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T12:35:00.928 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:00 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: self._run(expiration_interval) 2024-09-21T12:35:00.928 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:00 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T12:35:00.928 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:00 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T12:35:00.928 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:00 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T12:35:00.928 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:00 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T12:35:00.928 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:00 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T12:35:00.928 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:00 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: s = self.context.wrap_socket( 2024-09-21T12:35:00.928 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:00 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T12:35:00.928 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:00 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: return self.sslsocket_class._create( 2024-09-21T12:35:00.928 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:00 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T12:35:00.929 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:00 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: self.do_handshake() 2024-09-21T12:35:00.929 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:00 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T12:35:00.929 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:00 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: self._sslobj.do_handshake() 2024-09-21T12:35:00.929 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:00 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T12:35:00.929 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:00 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: 2024-09-21T12:35:01.887 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:01 smithi079 ceph-mon[23699]: [21/Sep/2024:12:35:00] ENGINE Bus STARTING 2024-09-21T12:35:01.887 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:01 smithi079 ceph-mon[23699]: [21/Sep/2024:12:35:00] ENGINE Serving on http://172.21.15.104:8765 2024-09-21T12:35:01.887 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:01 smithi079 ceph-mon[23699]: [21/Sep/2024:12:35:00] ENGINE Serving on https://172.21.15.104:7151 2024-09-21T12:35:01.887 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:01 smithi079 ceph-mon[23699]: [21/Sep/2024:12:35:00] ENGINE Bus STARTED 2024-09-21T12:35:01.887 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:01 smithi079 ceph-mon[23699]: [21/Sep/2024:12:35:00] ENGINE Error in HTTPServer.serve 2024-09-21T12:35:01.887 INFO:journalctl@ceph.mon.a.smithi079.stdout: Traceback (most recent call last): 2024-09-21T12:35:01.887 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T12:35:01.887 INFO:journalctl@ceph.mon.a.smithi079.stdout: self._connections.run(self.expiration_interval) 2024-09-21T12:35:01.887 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T12:35:01.888 INFO:journalctl@ceph.mon.a.smithi079.stdout: self._run(expiration_interval) 2024-09-21T12:35:01.888 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T12:35:01.888 INFO:journalctl@ceph.mon.a.smithi079.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T12:35:01.888 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T12:35:01.888 INFO:journalctl@ceph.mon.a.smithi079.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T12:35:01.888 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T12:35:01.888 INFO:journalctl@ceph.mon.a.smithi079.stdout: s = self.context.wrap_socket( 2024-09-21T12:35:01.888 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T12:35:01.888 INFO:journalctl@ceph.mon.a.smithi079.stdout: return self.sslsocket_class._create( 2024-09-21T12:35:01.888 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T12:35:01.888 INFO:journalctl@ceph.mon.a.smithi079.stdout: self.do_handshake() 2024-09-21T12:35:01.888 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T12:35:01.888 INFO:journalctl@ceph.mon.a.smithi079.stdout: self._sslobj.do_handshake() 2024-09-21T12:35:01.888 INFO:journalctl@ceph.mon.a.smithi079.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T12:35:01.888 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:01 smithi079 ceph-mon[23699]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 3 failed cephadm daemon(s)) 2024-09-21T12:35:01.889 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:01 smithi079 ceph-mon[23699]: Cluster is now healthy 2024-09-21T12:35:01.889 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:01 smithi079 ceph-mon[23699]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:35:01.889 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:01 smithi079 ceph-mon[23699]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi104", "name": "osd_memory_target"}]: dispatch 2024-09-21T12:35:01.889 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:01 smithi079 ceph-mon[23699]: from='mgr.25054 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi104", "name": "osd_memory_target"}]: dispatch 2024-09-21T12:35:01.889 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:01 smithi079 ceph-mon[23699]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:35:01.889 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:01 smithi079 ceph-mon[23699]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:35:01.889 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:01 smithi079 ceph-mon[23699]: Updating smithi079:/etc/ceph/ceph.conf 2024-09-21T12:35:01.889 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:01 smithi079 ceph-mon[23699]: Updating smithi104:/etc/ceph/ceph.conf 2024-09-21T12:35:01.889 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:01 smithi079 ceph-mon[23699]: from='client.? 172.21.15.79:0/3364742718' entity='client.iscsi.foo.smithi079.vlhrek' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:6800/1347188378"}]': finished 2024-09-21T12:35:01.889 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:01 smithi079 ceph-mon[23699]: osdmap e107: 8 total, 8 up, 8 in 2024-09-21T12:35:01.889 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:01 smithi079 ceph-mon[23699]: mgrmap e34: x(active, since 2s) 2024-09-21T12:35:01.889 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:01 smithi079 ceph-mon[23699]: from='mgr.25054 ' entity='mgr.x' 2024-09-21T12:35:01.889 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:01 smithi079 ceph-mon[23699]: from='mgr.25054 ' entity='mgr.x' 2024-09-21T12:35:01.889 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:01 smithi079 ceph-mon[23699]: from='mgr.25054 ' entity='mgr.x' 2024-09-21T12:35:01.890 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:01 smithi079 ceph-mon[30023]: [21/Sep/2024:12:35:00] ENGINE Bus STARTING 2024-09-21T12:35:01.890 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:01 smithi079 ceph-mon[30023]: [21/Sep/2024:12:35:00] ENGINE Serving on http://172.21.15.104:8765 2024-09-21T12:35:01.890 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:01 smithi079 ceph-mon[30023]: [21/Sep/2024:12:35:00] ENGINE Serving on https://172.21.15.104:7151 2024-09-21T12:35:01.890 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:01 smithi079 ceph-mon[30023]: [21/Sep/2024:12:35:00] ENGINE Bus STARTED 2024-09-21T12:35:01.890 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:01 smithi079 ceph-mon[30023]: [21/Sep/2024:12:35:00] ENGINE Error in HTTPServer.serve 2024-09-21T12:35:01.890 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2024-09-21T12:35:01.890 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T12:35:01.890 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._connections.run(self.expiration_interval) 2024-09-21T12:35:01.890 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T12:35:01.890 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._run(expiration_interval) 2024-09-21T12:35:01.890 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T12:35:01.890 INFO:journalctl@ceph.mon.c.smithi079.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T12:35:01.891 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T12:35:01.891 INFO:journalctl@ceph.mon.c.smithi079.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T12:35:01.891 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T12:35:01.891 INFO:journalctl@ceph.mon.c.smithi079.stdout: s = self.context.wrap_socket( 2024-09-21T12:35:01.891 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T12:35:01.891 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.sslsocket_class._create( 2024-09-21T12:35:01.891 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T12:35:01.891 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.do_handshake() 2024-09-21T12:35:01.891 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T12:35:01.891 INFO:journalctl@ceph.mon.c.smithi079.stdout: self._sslobj.do_handshake() 2024-09-21T12:35:01.891 INFO:journalctl@ceph.mon.c.smithi079.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T12:35:01.891 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:01 smithi079 ceph-mon[30023]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 3 failed cephadm daemon(s)) 2024-09-21T12:35:01.891 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:01 smithi079 ceph-mon[30023]: Cluster is now healthy 2024-09-21T12:35:01.891 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:01 smithi079 ceph-mon[30023]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:35:01.891 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:01 smithi079 ceph-mon[30023]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi104", "name": "osd_memory_target"}]: dispatch 2024-09-21T12:35:01.891 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:01 smithi079 ceph-mon[30023]: from='mgr.25054 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi104", "name": "osd_memory_target"}]: dispatch 2024-09-21T12:35:01.892 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:01 smithi079 ceph-mon[30023]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:35:01.892 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:01 smithi079 ceph-mon[30023]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:35:01.892 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:01 smithi079 ceph-mon[30023]: Updating smithi079:/etc/ceph/ceph.conf 2024-09-21T12:35:01.892 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:01 smithi079 ceph-mon[30023]: Updating smithi104:/etc/ceph/ceph.conf 2024-09-21T12:35:01.892 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:01 smithi079 ceph-mon[30023]: from='client.? 172.21.15.79:0/3364742718' entity='client.iscsi.foo.smithi079.vlhrek' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:6800/1347188378"}]': finished 2024-09-21T12:35:01.892 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:01 smithi079 ceph-mon[30023]: osdmap e107: 8 total, 8 up, 8 in 2024-09-21T12:35:01.892 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:01 smithi079 ceph-mon[30023]: mgrmap e34: x(active, since 2s) 2024-09-21T12:35:01.892 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:01 smithi079 ceph-mon[30023]: from='mgr.25054 ' entity='mgr.x' 2024-09-21T12:35:01.892 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:01 smithi079 ceph-mon[30023]: from='mgr.25054 ' entity='mgr.x' 2024-09-21T12:35:01.892 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:01 smithi079 ceph-mon[30023]: from='mgr.25054 ' entity='mgr.x' 2024-09-21T12:35:01.892 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:01 smithi079 ceph-mon[23699]: from='mgr.25054 ' entity='mgr.x' 2024-09-21T12:35:01.892 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:01 smithi079 ceph-mon[23699]: from='client.? 172.21.15.79:0/3872007533' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:0/3245174388"}]: dispatch 2024-09-21T12:35:01.893 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:01 smithi079 ceph-mon[23699]: from='client.? ' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:0/3245174388"}]: dispatch 2024-09-21T12:35:01.893 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:01 smithi079 ceph-mon[23699]: from='mgr.25054 ' entity='mgr.x' 2024-09-21T12:35:01.893 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:01 smithi079 ceph-mon[23699]: from='mgr.25054 ' entity='mgr.x' 2024-09-21T12:35:01.893 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:01 smithi079 ceph-mon[30023]: from='mgr.25054 ' entity='mgr.x' 2024-09-21T12:35:01.893 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:01 smithi079 ceph-mon[30023]: from='client.? 172.21.15.79:0/3872007533' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:0/3245174388"}]: dispatch 2024-09-21T12:35:01.893 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:01 smithi079 ceph-mon[30023]: from='client.? ' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:0/3245174388"}]: dispatch 2024-09-21T12:35:01.893 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:01 smithi079 ceph-mon[30023]: from='mgr.25054 ' entity='mgr.x' 2024-09-21T12:35:01.893 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:01 smithi079 ceph-mon[30023]: from='mgr.25054 ' entity='mgr.x' 2024-09-21T12:35:01.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:01 smithi104 ceph-mon[25639]: [21/Sep/2024:12:35:00] ENGINE Bus STARTING 2024-09-21T12:35:01.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:01 smithi104 ceph-mon[25639]: [21/Sep/2024:12:35:00] ENGINE Serving on http://172.21.15.104:8765 2024-09-21T12:35:01.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:01 smithi104 ceph-mon[25639]: [21/Sep/2024:12:35:00] ENGINE Serving on https://172.21.15.104:7151 2024-09-21T12:35:01.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:01 smithi104 ceph-mon[25639]: [21/Sep/2024:12:35:00] ENGINE Bus STARTED 2024-09-21T12:35:01.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:01 smithi104 ceph-mon[25639]: [21/Sep/2024:12:35:00] ENGINE Error in HTTPServer.serve 2024-09-21T12:35:01.928 INFO:journalctl@ceph.mon.b.smithi104.stdout: Traceback (most recent call last): 2024-09-21T12:35:01.928 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T12:35:01.928 INFO:journalctl@ceph.mon.b.smithi104.stdout: self._connections.run(self.expiration_interval) 2024-09-21T12:35:01.928 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T12:35:01.928 INFO:journalctl@ceph.mon.b.smithi104.stdout: self._run(expiration_interval) 2024-09-21T12:35:01.928 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T12:35:01.928 INFO:journalctl@ceph.mon.b.smithi104.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T12:35:01.928 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T12:35:01.928 INFO:journalctl@ceph.mon.b.smithi104.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T12:35:01.928 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T12:35:01.928 INFO:journalctl@ceph.mon.b.smithi104.stdout: s = self.context.wrap_socket( 2024-09-21T12:35:01.928 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T12:35:01.928 INFO:journalctl@ceph.mon.b.smithi104.stdout: return self.sslsocket_class._create( 2024-09-21T12:35:01.928 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T12:35:01.929 INFO:journalctl@ceph.mon.b.smithi104.stdout: self.do_handshake() 2024-09-21T12:35:01.929 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T12:35:01.929 INFO:journalctl@ceph.mon.b.smithi104.stdout: self._sslobj.do_handshake() 2024-09-21T12:35:01.929 INFO:journalctl@ceph.mon.b.smithi104.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T12:35:01.929 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:01 smithi104 ceph-mon[25639]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 3 failed cephadm daemon(s)) 2024-09-21T12:35:01.929 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:01 smithi104 ceph-mon[25639]: Cluster is now healthy 2024-09-21T12:35:01.929 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:01 smithi104 ceph-mon[25639]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:35:01.929 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:01 smithi104 ceph-mon[25639]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi104", "name": "osd_memory_target"}]: dispatch 2024-09-21T12:35:01.929 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:01 smithi104 ceph-mon[25639]: from='mgr.25054 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi104", "name": "osd_memory_target"}]: dispatch 2024-09-21T12:35:01.929 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:01 smithi104 ceph-mon[25639]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:35:01.929 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:01 smithi104 ceph-mon[25639]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:35:01.929 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:01 smithi104 ceph-mon[25639]: Updating smithi079:/etc/ceph/ceph.conf 2024-09-21T12:35:01.929 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:01 smithi104 ceph-mon[25639]: Updating smithi104:/etc/ceph/ceph.conf 2024-09-21T12:35:01.929 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:01 smithi104 ceph-mon[25639]: from='client.? 172.21.15.79:0/3364742718' entity='client.iscsi.foo.smithi079.vlhrek' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:6800/1347188378"}]': finished 2024-09-21T12:35:01.930 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:01 smithi104 ceph-mon[25639]: osdmap e107: 8 total, 8 up, 8 in 2024-09-21T12:35:01.930 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:01 smithi104 ceph-mon[25639]: mgrmap e34: x(active, since 2s) 2024-09-21T12:35:01.930 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:01 smithi104 ceph-mon[25639]: from='mgr.25054 ' entity='mgr.x' 2024-09-21T12:35:01.930 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:01 smithi104 ceph-mon[25639]: from='mgr.25054 ' entity='mgr.x' 2024-09-21T12:35:01.930 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:01 smithi104 ceph-mon[25639]: from='mgr.25054 ' entity='mgr.x' 2024-09-21T12:35:01.930 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:01 smithi104 ceph-mon[25639]: from='mgr.25054 ' entity='mgr.x' 2024-09-21T12:35:01.930 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:01 smithi104 ceph-mon[25639]: from='client.? 172.21.15.79:0/3872007533' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:0/3245174388"}]: dispatch 2024-09-21T12:35:01.930 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:01 smithi104 ceph-mon[25639]: from='client.? ' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:0/3245174388"}]: dispatch 2024-09-21T12:35:01.930 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:01 smithi104 ceph-mon[25639]: from='mgr.25054 ' entity='mgr.x' 2024-09-21T12:35:01.930 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:01 smithi104 ceph-mon[25639]: from='mgr.25054 ' entity='mgr.x' 2024-09-21T12:35:02.924 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:02 smithi079 ceph-mon[30023]: Updating smithi104:/var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T12:35:02.924 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:02 smithi079 ceph-mon[30023]: Updating smithi079:/var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T12:35:02.925 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:02 smithi079 ceph-mon[30023]: Updating smithi104:/etc/ceph/ceph.client.admin.keyring 2024-09-21T12:35:02.925 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:02 smithi079 ceph-mon[30023]: Updating smithi079:/etc/ceph/ceph.client.admin.keyring 2024-09-21T12:35:02.925 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:02 smithi079 ceph-mon[30023]: Updating smithi104:/var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T12:35:02.925 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:02 smithi079 ceph-mon[30023]: Updating smithi079:/var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T12:35:02.925 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:02 smithi079 ceph-mon[30023]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:35:02.925 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:02 smithi079 ceph-mon[30023]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-21T12:35:02.925 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:02 smithi079 ceph-mon[30023]: Health check failed: 2 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-09-21T12:35:02.925 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:02 smithi079 ceph-mon[30023]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi079", "caps": []}]: dispatch 2024-09-21T12:35:02.925 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:02 smithi079 ceph-mon[30023]: from='mgr.25054 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi079", "caps": []}]: dispatch 2024-09-21T12:35:02.926 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:02 smithi079 ceph-mon[30023]: from='mgr.25054 ' entity='mgr.x' 2024-09-21T12:35:02.926 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:02 smithi079 ceph-mon[30023]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi079.vlhrek", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-21T12:35:02.926 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:02 smithi079 ceph-mon[30023]: from='mgr.25054 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi079.vlhrek", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-21T12:35:02.926 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:02 smithi079 ceph-mon[30023]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:35:02.926 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:02 smithi079 ceph-mon[30023]: from='client.? ' entity='client.iscsi.foo.smithi079.vlhrek' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:0/3245174388"}]': finished 2024-09-21T12:35:02.926 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:02 smithi079 ceph-mon[30023]: osdmap e108: 8 total, 8 up, 8 in 2024-09-21T12:35:02.926 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:02 smithi079 ceph-mon[23699]: Updating smithi104:/var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T12:35:02.926 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:02 smithi079 ceph-mon[23699]: Updating smithi079:/var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T12:35:02.926 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:02 smithi079 ceph-mon[23699]: Updating smithi104:/etc/ceph/ceph.client.admin.keyring 2024-09-21T12:35:02.926 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:02 smithi079 ceph-mon[23699]: Updating smithi079:/etc/ceph/ceph.client.admin.keyring 2024-09-21T12:35:02.926 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:02 smithi079 ceph-mon[23699]: Updating smithi104:/var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T12:35:02.927 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:02 smithi079 ceph-mon[23699]: Updating smithi079:/var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T12:35:02.927 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:02 smithi079 ceph-mon[23699]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:35:02.927 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:02 smithi079 ceph-mon[23699]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-21T12:35:02.927 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:02 smithi079 ceph-mon[23699]: Health check failed: 2 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-09-21T12:35:02.927 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:02 smithi079 ceph-mon[23699]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi079", "caps": []}]: dispatch 2024-09-21T12:35:02.927 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:02 smithi079 ceph-mon[23699]: from='mgr.25054 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi079", "caps": []}]: dispatch 2024-09-21T12:35:02.927 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:02 smithi079 ceph-mon[23699]: from='mgr.25054 ' entity='mgr.x' 2024-09-21T12:35:02.927 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:02 smithi079 ceph-mon[23699]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi079.vlhrek", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-21T12:35:02.927 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:02 smithi079 ceph-mon[23699]: from='mgr.25054 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi079.vlhrek", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-21T12:35:02.927 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:02 smithi079 ceph-mon[23699]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:35:02.927 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:02 smithi079 ceph-mon[23699]: from='client.? ' entity='client.iscsi.foo.smithi079.vlhrek' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:0/3245174388"}]': finished 2024-09-21T12:35:02.927 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:02 smithi079 ceph-mon[23699]: osdmap e108: 8 total, 8 up, 8 in 2024-09-21T12:35:02.928 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:02 smithi104 ceph-mon[25639]: Updating smithi104:/var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T12:35:02.928 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:02 smithi104 ceph-mon[25639]: Updating smithi079:/var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T12:35:02.928 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:02 smithi104 ceph-mon[25639]: Updating smithi104:/etc/ceph/ceph.client.admin.keyring 2024-09-21T12:35:02.928 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:02 smithi104 ceph-mon[25639]: Updating smithi079:/etc/ceph/ceph.client.admin.keyring 2024-09-21T12:35:02.928 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:02 smithi104 ceph-mon[25639]: Updating smithi104:/var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T12:35:02.928 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:02 smithi104 ceph-mon[25639]: Updating smithi079:/var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T12:35:02.928 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:02 smithi104 ceph-mon[25639]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:35:02.928 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:02 smithi104 ceph-mon[25639]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-21T12:35:02.929 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:02 smithi104 ceph-mon[25639]: Health check failed: 2 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-09-21T12:35:02.929 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:02 smithi104 ceph-mon[25639]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi079", "caps": []}]: dispatch 2024-09-21T12:35:02.929 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:02 smithi104 ceph-mon[25639]: from='mgr.25054 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi079", "caps": []}]: dispatch 2024-09-21T12:35:02.929 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:02 smithi104 ceph-mon[25639]: from='mgr.25054 ' entity='mgr.x' 2024-09-21T12:35:02.929 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:02 smithi104 ceph-mon[25639]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi079.vlhrek", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-21T12:35:02.929 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:02 smithi104 ceph-mon[25639]: from='mgr.25054 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi079.vlhrek", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-21T12:35:02.929 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:02 smithi104 ceph-mon[25639]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:35:02.929 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:02 smithi104 ceph-mon[25639]: from='client.? ' entity='client.iscsi.foo.smithi079.vlhrek' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:0/3245174388"}]': finished 2024-09-21T12:35:02.929 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:02 smithi104 ceph-mon[25639]: osdmap e108: 8 total, 8 up, 8 in 2024-09-21T12:35:03.846 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:03 smithi104 ceph-mon[25639]: Reconfiguring iscsi.foo.smithi079.vlhrek (dependencies changed)... 2024-09-21T12:35:03.846 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:03 smithi104 ceph-mon[25639]: Reconfiguring daemon iscsi.foo.smithi079.vlhrek on smithi079 2024-09-21T12:35:03.846 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:03 smithi104 ceph-mon[25639]: from='mgr.25054 ' entity='mgr.x' 2024-09-21T12:35:03.847 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:03 smithi104 ceph-mon[25639]: from='mgr.25054 ' entity='mgr.x' 2024-09-21T12:35:03.847 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:03 smithi104 ceph-mon[25639]: from='mgr.25054 ' entity='mgr.x' 2024-09-21T12:35:03.847 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:03 smithi104 ceph-mon[25639]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:35:03.847 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:03 smithi104 ceph-mon[25639]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:35:03.847 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:03 smithi104 ceph-mon[25639]: mgrmap e35: x(active, since 4s) 2024-09-21T12:35:03.847 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:03 smithi104 ceph-mon[25639]: from='mgr.25054 ' entity='mgr.x' 2024-09-21T12:35:03.847 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:03 smithi104 ceph-mon[25639]: from='client.? 172.21.15.79:0/2421604914' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:0/1640215151"}]: dispatch 2024-09-21T12:35:03.847 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:03 smithi104 ceph-mon[25639]: from='client.? ' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:0/1640215151"}]: dispatch 2024-09-21T12:35:03.847 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:03 smithi104 ceph-mon[25639]: from='mgr.25054 ' entity='mgr.x' 2024-09-21T12:35:03.847 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:03 smithi104 ceph-mon[25639]: from='mgr.25054 ' entity='mgr.x' 2024-09-21T12:35:03.847 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:03 smithi104 ceph-mon[25639]: from='mgr.25054 ' entity='mgr.x' 2024-09-21T12:35:03.847 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:03 smithi104 ceph-mon[25639]: from='mgr.25054 ' entity='mgr.x' 2024-09-21T12:35:03.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:03 smithi079 ceph-mon[23699]: Reconfiguring iscsi.foo.smithi079.vlhrek (dependencies changed)... 2024-09-21T12:35:03.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:03 smithi079 ceph-mon[23699]: Reconfiguring daemon iscsi.foo.smithi079.vlhrek on smithi079 2024-09-21T12:35:03.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:03 smithi079 ceph-mon[23699]: from='mgr.25054 ' entity='mgr.x' 2024-09-21T12:35:03.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:03 smithi079 ceph-mon[23699]: from='mgr.25054 ' entity='mgr.x' 2024-09-21T12:35:03.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:03 smithi079 ceph-mon[23699]: from='mgr.25054 ' entity='mgr.x' 2024-09-21T12:35:03.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:03 smithi079 ceph-mon[23699]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:35:03.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:03 smithi079 ceph-mon[23699]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:35:03.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:03 smithi079 ceph-mon[23699]: mgrmap e35: x(active, since 4s) 2024-09-21T12:35:03.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:03 smithi079 ceph-mon[23699]: from='mgr.25054 ' entity='mgr.x' 2024-09-21T12:35:03.943 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:03 smithi079 ceph-mon[23699]: from='client.? 172.21.15.79:0/2421604914' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:0/1640215151"}]: dispatch 2024-09-21T12:35:03.943 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:03 smithi079 ceph-mon[23699]: from='client.? ' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:0/1640215151"}]: dispatch 2024-09-21T12:35:03.943 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:03 smithi079 ceph-mon[23699]: from='mgr.25054 ' entity='mgr.x' 2024-09-21T12:35:03.943 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:03 smithi079 ceph-mon[23699]: from='mgr.25054 ' entity='mgr.x' 2024-09-21T12:35:03.943 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:03 smithi079 ceph-mon[23699]: from='mgr.25054 ' entity='mgr.x' 2024-09-21T12:35:03.943 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:03 smithi079 ceph-mon[23699]: from='mgr.25054 ' entity='mgr.x' 2024-09-21T12:35:03.943 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:03 smithi079 ceph-mon[30023]: Reconfiguring iscsi.foo.smithi079.vlhrek (dependencies changed)... 2024-09-21T12:35:03.943 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:03 smithi079 ceph-mon[30023]: Reconfiguring daemon iscsi.foo.smithi079.vlhrek on smithi079 2024-09-21T12:35:03.943 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:03 smithi079 ceph-mon[30023]: from='mgr.25054 ' entity='mgr.x' 2024-09-21T12:35:03.943 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:03 smithi079 ceph-mon[30023]: from='mgr.25054 ' entity='mgr.x' 2024-09-21T12:35:03.944 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:03 smithi079 ceph-mon[30023]: from='mgr.25054 ' entity='mgr.x' 2024-09-21T12:35:03.944 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:03 smithi079 ceph-mon[30023]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:35:03.944 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:03 smithi079 ceph-mon[30023]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:35:03.944 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:03 smithi079 ceph-mon[30023]: mgrmap e35: x(active, since 4s) 2024-09-21T12:35:03.944 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:03 smithi079 ceph-mon[30023]: from='mgr.25054 ' entity='mgr.x' 2024-09-21T12:35:03.944 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:03 smithi079 ceph-mon[30023]: from='client.? 172.21.15.79:0/2421604914' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:0/1640215151"}]: dispatch 2024-09-21T12:35:03.944 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:03 smithi079 ceph-mon[30023]: from='client.? ' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:0/1640215151"}]: dispatch 2024-09-21T12:35:03.944 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:03 smithi079 ceph-mon[30023]: from='mgr.25054 ' entity='mgr.x' 2024-09-21T12:35:03.944 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:03 smithi079 ceph-mon[30023]: from='mgr.25054 ' entity='mgr.x' 2024-09-21T12:35:03.944 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:03 smithi079 ceph-mon[30023]: from='mgr.25054 ' entity='mgr.x' 2024-09-21T12:35:03.944 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:03 smithi079 ceph-mon[30023]: from='mgr.25054 ' entity='mgr.x' 2024-09-21T12:35:04.238 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:35:04 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[70883]: [21/Sep/2024:12:35:04] ENGINE Bus STARTING 2024-09-21T12:35:04.238 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:35:04 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[70883]: CherryPy Checker: 2024-09-21T12:35:04.238 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:35:04 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[70883]: The Application mounted at '' has an empty config. 2024-09-21T12:35:04.238 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:35:04 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[70883]: 2024-09-21T12:35:04.238 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:35:04 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[70883]: [21/Sep/2024:12:35:04] ENGINE Serving on http://:::9283 2024-09-21T12:35:04.238 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:35:04 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[70883]: [21/Sep/2024:12:35:04] ENGINE Bus STARTED 2024-09-21T12:35:05.036 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:04 smithi079 ceph-mon[30023]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:35:05.036 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:04 smithi079 ceph-mon[30023]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:35:05.036 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:04 smithi079 ceph-mon[30023]: from='client.? ' entity='client.iscsi.foo.smithi079.vlhrek' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:0/1640215151"}]': finished 2024-09-21T12:35:05.037 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:04 smithi079 ceph-mon[30023]: osdmap e109: 8 total, 8 up, 8 in 2024-09-21T12:35:05.037 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:04 smithi079 ceph-mon[30023]: Standby manager daemon y started 2024-09-21T12:35:05.037 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:04 smithi079 ceph-mon[30023]: from='mgr.? 172.21.15.79:0/2234600007' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-21T12:35:05.037 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:04 smithi079 ceph-mon[30023]: from='mgr.? 172.21.15.79:0/2234600007' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-21T12:35:05.037 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:04 smithi079 ceph-mon[30023]: from='mgr.? 172.21.15.79:0/2234600007' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-09-21T12:35:05.037 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:04 smithi079 ceph-mon[30023]: from='mgr.? 172.21.15.79:0/2234600007' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-21T12:35:05.037 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:04 smithi079 ceph-mon[30023]: from='client.? 172.21.15.79:0/1559745417' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:0/1452502078"}]: dispatch 2024-09-21T12:35:05.037 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:04 smithi079 ceph-mon[23699]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:35:05.037 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:04 smithi079 ceph-mon[23699]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:35:05.037 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:04 smithi079 ceph-mon[23699]: from='client.? ' entity='client.iscsi.foo.smithi079.vlhrek' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:0/1640215151"}]': finished 2024-09-21T12:35:05.037 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:04 smithi079 ceph-mon[23699]: osdmap e109: 8 total, 8 up, 8 in 2024-09-21T12:35:05.037 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:04 smithi079 ceph-mon[23699]: Standby manager daemon y started 2024-09-21T12:35:05.037 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:04 smithi079 ceph-mon[23699]: from='mgr.? 172.21.15.79:0/2234600007' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-21T12:35:05.038 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:04 smithi079 ceph-mon[23699]: from='mgr.? 172.21.15.79:0/2234600007' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-21T12:35:05.038 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:04 smithi079 ceph-mon[23699]: from='mgr.? 172.21.15.79:0/2234600007' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-09-21T12:35:05.038 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:04 smithi079 ceph-mon[23699]: from='mgr.? 172.21.15.79:0/2234600007' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-21T12:35:05.038 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:04 smithi079 ceph-mon[23699]: from='client.? 172.21.15.79:0/1559745417' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:0/1452502078"}]: dispatch 2024-09-21T12:35:05.169 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:04 smithi104 ceph-mon[25639]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:35:05.169 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:04 smithi104 ceph-mon[25639]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 254 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:35:05.170 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:04 smithi104 ceph-mon[25639]: from='client.? ' entity='client.iscsi.foo.smithi079.vlhrek' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:0/1640215151"}]': finished 2024-09-21T12:35:05.170 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:04 smithi104 ceph-mon[25639]: osdmap e109: 8 total, 8 up, 8 in 2024-09-21T12:35:05.170 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:04 smithi104 ceph-mon[25639]: Standby manager daemon y started 2024-09-21T12:35:05.170 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:04 smithi104 ceph-mon[25639]: from='mgr.? 172.21.15.79:0/2234600007' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-21T12:35:05.170 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:04 smithi104 ceph-mon[25639]: from='mgr.? 172.21.15.79:0/2234600007' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-21T12:35:05.170 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:04 smithi104 ceph-mon[25639]: from='mgr.? 172.21.15.79:0/2234600007' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-09-21T12:35:05.170 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:04 smithi104 ceph-mon[25639]: from='mgr.? 172.21.15.79:0/2234600007' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-21T12:35:05.170 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:04 smithi104 ceph-mon[25639]: from='client.? 172.21.15.79:0/1559745417' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:0/1452502078"}]: dispatch 2024-09-21T12:35:06.143 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:05 smithi079 ceph-mon[23699]: from='client.? 172.21.15.79:0/1559745417' entity='client.iscsi.foo.smithi079.vlhrek' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:0/1452502078"}]': finished 2024-09-21T12:35:06.143 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:05 smithi079 ceph-mon[23699]: osdmap e110: 8 total, 8 up, 8 in 2024-09-21T12:35:06.143 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:05 smithi079 ceph-mon[23699]: mgrmap e36: x(active, since 6s), standbys: y 2024-09-21T12:35:06.143 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:05 smithi079 ceph-mon[23699]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-21T12:35:06.143 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:05 smithi079 ceph-mon[23699]: from='client.? 172.21.15.79:0/2600193282' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:0/2455578327"}]: dispatch 2024-09-21T12:35:06.143 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:05 smithi079 ceph-mon[23699]: from='client.? ' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:0/2455578327"}]: dispatch 2024-09-21T12:35:06.144 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:05 smithi079 ceph-mon[30023]: from='client.? 172.21.15.79:0/1559745417' entity='client.iscsi.foo.smithi079.vlhrek' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:0/1452502078"}]': finished 2024-09-21T12:35:06.144 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:05 smithi079 ceph-mon[30023]: osdmap e110: 8 total, 8 up, 8 in 2024-09-21T12:35:06.144 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:05 smithi079 ceph-mon[30023]: mgrmap e36: x(active, since 6s), standbys: y 2024-09-21T12:35:06.144 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:05 smithi079 ceph-mon[30023]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-21T12:35:06.144 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:05 smithi079 ceph-mon[30023]: from='client.? 172.21.15.79:0/2600193282' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:0/2455578327"}]: dispatch 2024-09-21T12:35:06.144 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:05 smithi079 ceph-mon[30023]: from='client.? ' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:0/2455578327"}]: dispatch 2024-09-21T12:35:06.176 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:05 smithi104 ceph-mon[25639]: from='client.? 172.21.15.79:0/1559745417' entity='client.iscsi.foo.smithi079.vlhrek' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:0/1452502078"}]': finished 2024-09-21T12:35:06.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:05 smithi104 ceph-mon[25639]: osdmap e110: 8 total, 8 up, 8 in 2024-09-21T12:35:06.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:05 smithi104 ceph-mon[25639]: mgrmap e36: x(active, since 6s), standbys: y 2024-09-21T12:35:06.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:05 smithi104 ceph-mon[25639]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-21T12:35:06.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:05 smithi104 ceph-mon[25639]: from='client.? 172.21.15.79:0/2600193282' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:0/2455578327"}]: dispatch 2024-09-21T12:35:06.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:05 smithi104 ceph-mon[25639]: from='client.? ' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:0/2455578327"}]: dispatch 2024-09-21T12:35:07.022 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:06 smithi104 ceph-mon[25639]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 84 KiB/s rd, 0 B/s wr, 35 op/s 2024-09-21T12:35:07.023 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:06 smithi104 ceph-mon[25639]: from='client.? ' entity='client.iscsi.foo.smithi079.vlhrek' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:0/2455578327"}]': finished 2024-09-21T12:35:07.023 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:06 smithi104 ceph-mon[25639]: osdmap e111: 8 total, 8 up, 8 in 2024-09-21T12:35:07.023 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:06 smithi104 ceph-mon[25639]: Health check update: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-09-21T12:35:07.026 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:06 smithi079 ceph-mon[30023]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 84 KiB/s rd, 0 B/s wr, 35 op/s 2024-09-21T12:35:07.026 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:06 smithi079 ceph-mon[30023]: from='client.? ' entity='client.iscsi.foo.smithi079.vlhrek' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:0/2455578327"}]': finished 2024-09-21T12:35:07.026 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:06 smithi079 ceph-mon[30023]: osdmap e111: 8 total, 8 up, 8 in 2024-09-21T12:35:07.026 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:06 smithi079 ceph-mon[30023]: Health check update: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-09-21T12:35:07.026 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:06 smithi079 ceph-mon[23699]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 84 KiB/s rd, 0 B/s wr, 35 op/s 2024-09-21T12:35:07.026 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:06 smithi079 ceph-mon[23699]: from='client.? ' entity='client.iscsi.foo.smithi079.vlhrek' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.79:0/2455578327"}]': finished 2024-09-21T12:35:07.026 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:06 smithi079 ceph-mon[23699]: osdmap e111: 8 total, 8 up, 8 in 2024-09-21T12:35:07.026 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:06 smithi079 ceph-mon[23699]: Health check update: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-09-21T12:35:09.143 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:08 smithi079 ceph-mon[30023]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 45 KiB/s rd, 0 B/s wr, 19 op/s 2024-09-21T12:35:09.143 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:08 smithi079 ceph-mon[30023]: from='mgr.25054 ' entity='mgr.x' 2024-09-21T12:35:09.143 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:08 smithi079 ceph-mon[30023]: from='mgr.25054 ' entity='mgr.x' 2024-09-21T12:35:09.143 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:08 smithi079 ceph-mon[30023]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 15 op/s 2024-09-21T12:35:09.143 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:08 smithi079 ceph-mon[30023]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 45 KiB/s rd, 0 B/s wr, 19 op/s 2024-09-21T12:35:09.143 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:08 smithi079 ceph-mon[30023]: from='mgr.25054 ' entity='mgr.x' 2024-09-21T12:35:09.143 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:08 smithi079 ceph-mon[23699]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 45 KiB/s rd, 0 B/s wr, 19 op/s 2024-09-21T12:35:09.143 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:08 smithi079 ceph-mon[23699]: from='mgr.25054 ' entity='mgr.x' 2024-09-21T12:35:09.144 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:08 smithi079 ceph-mon[23699]: from='mgr.25054 ' entity='mgr.x' 2024-09-21T12:35:09.144 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:08 smithi079 ceph-mon[23699]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 15 op/s 2024-09-21T12:35:09.144 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:08 smithi079 ceph-mon[23699]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 45 KiB/s rd, 0 B/s wr, 19 op/s 2024-09-21T12:35:09.144 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:08 smithi079 ceph-mon[23699]: from='mgr.25054 ' entity='mgr.x' 2024-09-21T12:35:09.176 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:08 smithi104 ceph-mon[25639]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 45 KiB/s rd, 0 B/s wr, 19 op/s 2024-09-21T12:35:09.176 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:08 smithi104 ceph-mon[25639]: from='mgr.25054 ' entity='mgr.x' 2024-09-21T12:35:09.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:08 smithi104 ceph-mon[25639]: from='mgr.25054 ' entity='mgr.x' 2024-09-21T12:35:09.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:08 smithi104 ceph-mon[25639]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 15 op/s 2024-09-21T12:35:09.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:08 smithi104 ceph-mon[25639]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 45 KiB/s rd, 0 B/s wr, 19 op/s 2024-09-21T12:35:09.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:08 smithi104 ceph-mon[25639]: from='mgr.25054 ' entity='mgr.x' 2024-09-21T12:35:10.014 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:09 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: 2024-09-21T12:35:09.706+0000 7ff086688640 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi079.vlhrek ... 2024-09-21T12:35:10.014 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:09 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: Creating ceph-iscsi config... 2024-09-21T12:35:10.014 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:09 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: Write file: /var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/iscsi.foo.smithi079.vlhrek/iscsi-gateway.cfg 2024-09-21T12:35:10.014 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:09 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: Write file: /var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/iscsi.foo.smithi079.vlhrek/tcmu-runner-entrypoint.sh 2024-09-21T12:35:10.014 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:09 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: Non-zero exit code 1 from systemctl restart ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek 2024-09-21T12:35:10.014 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:09 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: systemctl: stderr Job for ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service failed because the control process exited with error code. 2024-09-21T12:35:10.015 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:09 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: systemctl: stderr See "systemctl status ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service" and "journalctl -xeu ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service" for details. 2024-09-21T12:35:10.015 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:09 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: Traceback (most recent call last): 2024-09-21T12:35:10.015 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:09 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-21T12:35:10.015 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:09 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: return _run_code(code, main_globals, None, 2024-09-21T12:35:10.015 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:09 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-21T12:35:10.015 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:09 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: exec(code, run_globals) 2024-09-21T12:35:10.015 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:09 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 5628, in 2024-09-21T12:35:10.015 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:09 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 5616, in main 2024-09-21T12:35:10.015 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:09 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 3090, in command_deploy_from 2024-09-21T12:35:10.015 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:09 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 3125, in _common_deploy 2024-09-21T12:35:10.015 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:09 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 3145, in _deploy_daemon_container 2024-09-21T12:35:10.015 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:09 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 1147, in deploy_daemon 2024-09-21T12:35:10.015 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:09 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: File "/tmp/tmptpwpfqo2.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-21T12:35:10.015 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:09 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: RuntimeError: Failed command: systemctl restart ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek: Job for ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service failed because the control process exited with error code. 2024-09-21T12:35:10.016 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:09 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: See "systemctl status ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service" and "journalctl -xeu ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service" for details. 2024-09-21T12:35:10.016 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:09 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: Traceback (most recent call last): 2024-09-21T12:35:10.016 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:09 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1164, in _check_daemons 2024-09-21T12:35:10.016 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:09 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: self.mgr._daemon_action(daemon_spec, action=action) 2024-09-21T12:35:10.016 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:09 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2610, in _daemon_action 2024-09-21T12:35:10.016 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:09 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: return self.wait_async( 2024-09-21T12:35:10.016 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:09 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: File "/usr/share/ceph/mgr/cephadm/module.py", line 878, in wait_async 2024-09-21T12:35:10.016 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:09 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: return self.event_loop.get_result(coro, timeout) 2024-09-21T12:35:10.017 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:09 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-09-21T12:35:10.017 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:09 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: return future.result(timeout) 2024-09-21T12:35:10.017 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:09 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: File "/lib64/python3.9/concurrent/futures/_base.py", line 446, in result 2024-09-21T12:35:10.017 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:09 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: return self.__get_result() 2024-09-21T12:35:10.017 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:09 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: File "/lib64/python3.9/concurrent/futures/_base.py", line 391, in __get_result 2024-09-21T12:35:10.017 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:09 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: raise self._exception 2024-09-21T12:35:10.017 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:09 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1400, in _create_daemon 2024-09-21T12:35:10.017 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:09 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: out, err, code = await self._run_cephadm( 2024-09-21T12:35:10.017 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:09 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1743, in _run_cephadm 2024-09-21T12:35:10.017 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:09 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: raise OrchestratorError( 2024-09-21T12:35:10.017 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:09 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi079.vlhrek ... 2024-09-21T12:35:10.017 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:09 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: Creating ceph-iscsi config... 2024-09-21T12:35:10.017 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:09 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: Write file: /var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/iscsi.foo.smithi079.vlhrek/iscsi-gateway.cfg 2024-09-21T12:35:10.017 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:09 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: Write file: /var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/iscsi.foo.smithi079.vlhrek/tcmu-runner-entrypoint.sh 2024-09-21T12:35:10.017 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:09 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: Non-zero exit code 1 from systemctl restart ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek 2024-09-21T12:35:10.018 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:09 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: systemctl: stderr Job for ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service failed because the control process exited with error code. 2024-09-21T12:35:10.018 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:09 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: systemctl: stderr See "systemctl status ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service" and "journalctl -xeu ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service" for details. 2024-09-21T12:35:10.018 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:09 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: Traceback (most recent call last): 2024-09-21T12:35:10.018 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:09 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-21T12:35:10.018 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:09 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: return _run_code(code, main_globals, None, 2024-09-21T12:35:10.018 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:09 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-21T12:35:10.018 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:09 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: exec(code, run_globals) 2024-09-21T12:35:10.018 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:09 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 5628, in 2024-09-21T12:35:10.018 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:09 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 5616, in main 2024-09-21T12:35:10.018 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:09 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 3090, in command_deploy_from 2024-09-21T12:35:10.018 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:09 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 3125, in _common_deploy 2024-09-21T12:35:10.018 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:09 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 3145, in _deploy_daemon_container 2024-09-21T12:35:10.018 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:09 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 1147, in deploy_daemon 2024-09-21T12:35:10.019 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:09 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: File "/tmp/tmptpwpfqo2.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-21T12:35:10.019 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:09 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: RuntimeError: Failed command: systemctl restart ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek: Job for ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service failed because the control process exited with error code. 2024-09-21T12:35:10.019 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:09 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: See "systemctl status ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service" and "journalctl -xeu ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service" for details. 2024-09-21T12:35:10.268 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:10 smithi079 ceph-mon[23699]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi079.vlhrek ... 2024-09-21T12:35:10.268 INFO:journalctl@ceph.mon.a.smithi079.stdout: Creating ceph-iscsi config... 2024-09-21T12:35:10.268 INFO:journalctl@ceph.mon.a.smithi079.stdout: Write file: /var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/iscsi.foo.smithi079.vlhrek/iscsi-gateway.cfg 2024-09-21T12:35:10.268 INFO:journalctl@ceph.mon.a.smithi079.stdout: Write file: /var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/iscsi.foo.smithi079.vlhrek/tcmu-runner-entrypoint.sh 2024-09-21T12:35:10.268 INFO:journalctl@ceph.mon.a.smithi079.stdout: Non-zero exit code 1 from systemctl restart ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek 2024-09-21T12:35:10.268 INFO:journalctl@ceph.mon.a.smithi079.stdout: systemctl: stderr Job for ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service failed because the control process exited with error code. 2024-09-21T12:35:10.268 INFO:journalctl@ceph.mon.a.smithi079.stdout: systemctl: stderr See "systemctl status ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service" and "journalctl -xeu ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service" for details. 2024-09-21T12:35:10.269 INFO:journalctl@ceph.mon.a.smithi079.stdout: Traceback (most recent call last): 2024-09-21T12:35:10.269 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-21T12:35:10.269 INFO:journalctl@ceph.mon.a.smithi079.stdout: return _run_code(code, main_globals, None, 2024-09-21T12:35:10.269 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-21T12:35:10.269 INFO:journalctl@ceph.mon.a.smithi079.stdout: exec(code, run_globals) 2024-09-21T12:35:10.269 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 5628, in 2024-09-21T12:35:10.269 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 5616, in main 2024-09-21T12:35:10.269 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 3090, in command_deploy_from 2024-09-21T12:35:10.269 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 3125, in _common_deploy 2024-09-21T12:35:10.269 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 3145, in _deploy_daemon_container 2024-09-21T12:35:10.269 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 1147, in deploy_daemon 2024-09-21T12:35:10.269 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/tmp/tmptpwpfqo2.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-21T12:35:10.269 INFO:journalctl@ceph.mon.a.smithi079.stdout: RuntimeError: Failed command: systemctl restart ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek: Job for ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service failed because the control process exited with error code. 2024-09-21T12:35:10.269 INFO:journalctl@ceph.mon.a.smithi079.stdout: See "systemctl status ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service" and "journalctl -xeu ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service" for details. 2024-09-21T12:35:10.270 INFO:journalctl@ceph.mon.a.smithi079.stdout: Traceback (most recent call last): 2024-09-21T12:35:10.270 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1164, in _check_daemons 2024-09-21T12:35:10.270 INFO:journalctl@ceph.mon.a.smithi079.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-09-21T12:35:10.270 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2610, in _daemon_action 2024-09-21T12:35:10.270 INFO:journalctl@ceph.mon.a.smithi079.stdout: return self.wait_async( 2024-09-21T12:35:10.270 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 878, in wait_async 2024-09-21T12:35:10.270 INFO:journalctl@ceph.mon.a.smithi079.stdout: return self.event_loop.get_result(coro, timeout) 2024-09-21T12:35:10.270 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-09-21T12:35:10.270 INFO:journalctl@ceph.mon.a.smithi079.stdout: return future.result(timeout) 2024-09-21T12:35:10.270 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/lib64/python3.9/concurrent/futures/_base.py", line 446, in result 2024-09-21T12:35:10.270 INFO:journalctl@ceph.mon.a.smithi079.stdout: return self.__get_result() 2024-09-21T12:35:10.270 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/lib64/python3.9/concurrent/futures/_base.py", line 391, in __get_result 2024-09-21T12:35:10.270 INFO:journalctl@ceph.mon.a.smithi079.stdout: raise self._exception 2024-09-21T12:35:10.270 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1400, in _create_daemon 2024-09-21T12:35:10.271 INFO:journalctl@ceph.mon.a.smithi079.stdout: out, err, code = await self._run_cephadm( 2024-09-21T12:35:10.271 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1743, in _run_cephadm 2024-09-21T12:35:10.271 INFO:journalctl@ceph.mon.a.smithi079.stdout: raise OrchestratorError( 2024-09-21T12:35:10.271 INFO:journalctl@ceph.mon.a.smithi079.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi079.vlhrek ... 2024-09-21T12:35:10.271 INFO:journalctl@ceph.mon.a.smithi079.stdout: Creating ceph-iscsi config... 2024-09-21T12:35:10.271 INFO:journalctl@ceph.mon.a.smithi079.stdout: Write file: /var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/iscsi.foo.smithi079.vlhrek/iscsi-gateway.cfg 2024-09-21T12:35:10.271 INFO:journalctl@ceph.mon.a.smithi079.stdout: Write file: /var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/iscsi.foo.smithi079.vlhrek/tcmu-runner-entrypoint.sh 2024-09-21T12:35:10.271 INFO:journalctl@ceph.mon.a.smithi079.stdout: Non-zero exit code 1 from systemctl restart ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek 2024-09-21T12:35:10.271 INFO:journalctl@ceph.mon.a.smithi079.stdout: systemctl: stderr Job for ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service failed because the control process exited with error code. 2024-09-21T12:35:10.271 INFO:journalctl@ceph.mon.a.smithi079.stdout: systemctl: stderr See "systemctl status ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service" and "journalctl -xeu ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service" for details. 2024-09-21T12:35:10.271 INFO:journalctl@ceph.mon.a.smithi079.stdout: Traceback (most recent call last): 2024-09-21T12:35:10.271 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-21T12:35:10.271 INFO:journalctl@ceph.mon.a.smithi079.stdout: return _run_code(code, main_globals, None, 2024-09-21T12:35:10.271 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-21T12:35:10.271 INFO:journalctl@ceph.mon.a.smithi079.stdout: exec(code, run_globals) 2024-09-21T12:35:10.271 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 5628, in 2024-09-21T12:35:10.272 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 5616, in main 2024-09-21T12:35:10.272 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 3090, in command_deploy_from 2024-09-21T12:35:10.272 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 3125, in _common_deploy 2024-09-21T12:35:10.272 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 3145, in _deploy_daemon_container 2024-09-21T12:35:10.272 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 1147, in deploy_daemon 2024-09-21T12:35:10.272 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/tmp/tmptpwpfqo2.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-21T12:35:10.272 INFO:journalctl@ceph.mon.a.smithi079.stdout: RuntimeError: Failed command: systemctl restart ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek: Job for ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service failed because the control process exited with error code. 2024-09-21T12:35:10.272 INFO:journalctl@ceph.mon.a.smithi079.stdout: See "systemctl status ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service" and "journalctl -xeu ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service" for details. 2024-09-21T12:35:10.272 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:10 smithi079 ceph-mon[23699]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-21T12:35:10.272 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:10 smithi079 ceph-mon[23699]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 821 B/s rd, 1 op/s 2024-09-21T12:35:10.272 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:10 smithi079 ceph-mon[23699]: Reconfiguring daemon prometheus.a on smithi104 2024-09-21T12:35:10.273 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:10 smithi079 ceph-mon[30023]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi079.vlhrek ... 2024-09-21T12:35:10.273 INFO:journalctl@ceph.mon.c.smithi079.stdout: Creating ceph-iscsi config... 2024-09-21T12:35:10.273 INFO:journalctl@ceph.mon.c.smithi079.stdout: Write file: /var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/iscsi.foo.smithi079.vlhrek/iscsi-gateway.cfg 2024-09-21T12:35:10.273 INFO:journalctl@ceph.mon.c.smithi079.stdout: Write file: /var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/iscsi.foo.smithi079.vlhrek/tcmu-runner-entrypoint.sh 2024-09-21T12:35:10.273 INFO:journalctl@ceph.mon.c.smithi079.stdout: Non-zero exit code 1 from systemctl restart ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek 2024-09-21T12:35:10.273 INFO:journalctl@ceph.mon.c.smithi079.stdout: systemctl: stderr Job for ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service failed because the control process exited with error code. 2024-09-21T12:35:10.273 INFO:journalctl@ceph.mon.c.smithi079.stdout: systemctl: stderr See "systemctl status ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service" and "journalctl -xeu ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service" for details. 2024-09-21T12:35:10.273 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2024-09-21T12:35:10.273 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-21T12:35:10.273 INFO:journalctl@ceph.mon.c.smithi079.stdout: return _run_code(code, main_globals, None, 2024-09-21T12:35:10.274 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-21T12:35:10.274 INFO:journalctl@ceph.mon.c.smithi079.stdout: exec(code, run_globals) 2024-09-21T12:35:10.274 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 5628, in 2024-09-21T12:35:10.274 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 5616, in main 2024-09-21T12:35:10.274 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 3090, in command_deploy_from 2024-09-21T12:35:10.274 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 3125, in _common_deploy 2024-09-21T12:35:10.274 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 3145, in _deploy_daemon_container 2024-09-21T12:35:10.274 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 1147, in deploy_daemon 2024-09-21T12:35:10.274 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/tmp/tmptpwpfqo2.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-21T12:35:10.274 INFO:journalctl@ceph.mon.c.smithi079.stdout: RuntimeError: Failed command: systemctl restart ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek: Job for ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service failed because the control process exited with error code. 2024-09-21T12:35:10.274 INFO:journalctl@ceph.mon.c.smithi079.stdout: See "systemctl status ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service" and "journalctl -xeu ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service" for details. 2024-09-21T12:35:10.274 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2024-09-21T12:35:10.275 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1164, in _check_daemons 2024-09-21T12:35:10.275 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-09-21T12:35:10.275 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2610, in _daemon_action 2024-09-21T12:35:10.275 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.wait_async( 2024-09-21T12:35:10.275 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 878, in wait_async 2024-09-21T12:35:10.275 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro, timeout) 2024-09-21T12:35:10.275 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-09-21T12:35:10.275 INFO:journalctl@ceph.mon.c.smithi079.stdout: return future.result(timeout) 2024-09-21T12:35:10.275 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.9/concurrent/futures/_base.py", line 446, in result 2024-09-21T12:35:10.275 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2024-09-21T12:35:10.275 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.9/concurrent/futures/_base.py", line 391, in __get_result 2024-09-21T12:35:10.275 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2024-09-21T12:35:10.275 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1400, in _create_daemon 2024-09-21T12:35:10.275 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._run_cephadm( 2024-09-21T12:35:10.276 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1743, in _run_cephadm 2024-09-21T12:35:10.276 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError( 2024-09-21T12:35:10.276 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi079.vlhrek ... 2024-09-21T12:35:10.276 INFO:journalctl@ceph.mon.c.smithi079.stdout: Creating ceph-iscsi config... 2024-09-21T12:35:10.276 INFO:journalctl@ceph.mon.c.smithi079.stdout: Write file: /var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/iscsi.foo.smithi079.vlhrek/iscsi-gateway.cfg 2024-09-21T12:35:10.276 INFO:journalctl@ceph.mon.c.smithi079.stdout: Write file: /var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/iscsi.foo.smithi079.vlhrek/tcmu-runner-entrypoint.sh 2024-09-21T12:35:10.276 INFO:journalctl@ceph.mon.c.smithi079.stdout: Non-zero exit code 1 from systemctl restart ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek 2024-09-21T12:35:10.276 INFO:journalctl@ceph.mon.c.smithi079.stdout: systemctl: stderr Job for ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service failed because the control process exited with error code. 2024-09-21T12:35:10.276 INFO:journalctl@ceph.mon.c.smithi079.stdout: systemctl: stderr See "systemctl status ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service" and "journalctl -xeu ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service" for details. 2024-09-21T12:35:10.276 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2024-09-21T12:35:10.276 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-21T12:35:10.276 INFO:journalctl@ceph.mon.c.smithi079.stdout: return _run_code(code, main_globals, None, 2024-09-21T12:35:10.276 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-21T12:35:10.276 INFO:journalctl@ceph.mon.c.smithi079.stdout: exec(code, run_globals) 2024-09-21T12:35:10.277 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 5628, in 2024-09-21T12:35:10.277 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 5616, in main 2024-09-21T12:35:10.277 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 3090, in command_deploy_from 2024-09-21T12:35:10.277 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 3125, in _common_deploy 2024-09-21T12:35:10.277 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 3145, in _deploy_daemon_container 2024-09-21T12:35:10.277 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 1147, in deploy_daemon 2024-09-21T12:35:10.277 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/tmp/tmptpwpfqo2.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-21T12:35:10.277 INFO:journalctl@ceph.mon.c.smithi079.stdout: RuntimeError: Failed command: systemctl restart ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek: Job for ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service failed because the control process exited with error code. 2024-09-21T12:35:10.277 INFO:journalctl@ceph.mon.c.smithi079.stdout: See "systemctl status ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service" and "journalctl -xeu ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service" for details. 2024-09-21T12:35:10.277 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:10 smithi079 ceph-mon[30023]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-21T12:35:10.277 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:10 smithi079 ceph-mon[30023]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 821 B/s rd, 1 op/s 2024-09-21T12:35:10.277 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:10 smithi079 ceph-mon[30023]: Reconfiguring daemon prometheus.a on smithi104 2024-09-21T12:35:10.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:10 smithi104 ceph-mon[25639]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi079.vlhrek ... 2024-09-21T12:35:10.427 INFO:journalctl@ceph.mon.b.smithi104.stdout: Creating ceph-iscsi config... 2024-09-21T12:35:10.427 INFO:journalctl@ceph.mon.b.smithi104.stdout: Write file: /var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/iscsi.foo.smithi079.vlhrek/iscsi-gateway.cfg 2024-09-21T12:35:10.427 INFO:journalctl@ceph.mon.b.smithi104.stdout: Write file: /var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/iscsi.foo.smithi079.vlhrek/tcmu-runner-entrypoint.sh 2024-09-21T12:35:10.427 INFO:journalctl@ceph.mon.b.smithi104.stdout: Non-zero exit code 1 from systemctl restart ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek 2024-09-21T12:35:10.427 INFO:journalctl@ceph.mon.b.smithi104.stdout: systemctl: stderr Job for ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service failed because the control process exited with error code. 2024-09-21T12:35:10.427 INFO:journalctl@ceph.mon.b.smithi104.stdout: systemctl: stderr See "systemctl status ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service" and "journalctl -xeu ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service" for details. 2024-09-21T12:35:10.427 INFO:journalctl@ceph.mon.b.smithi104.stdout: Traceback (most recent call last): 2024-09-21T12:35:10.428 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-21T12:35:10.428 INFO:journalctl@ceph.mon.b.smithi104.stdout: return _run_code(code, main_globals, None, 2024-09-21T12:35:10.428 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-21T12:35:10.428 INFO:journalctl@ceph.mon.b.smithi104.stdout: exec(code, run_globals) 2024-09-21T12:35:10.428 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 5628, in 2024-09-21T12:35:10.428 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 5616, in main 2024-09-21T12:35:10.428 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 3090, in command_deploy_from 2024-09-21T12:35:10.428 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 3125, in _common_deploy 2024-09-21T12:35:10.428 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 3145, in _deploy_daemon_container 2024-09-21T12:35:10.428 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 1147, in deploy_daemon 2024-09-21T12:35:10.428 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/tmp/tmptpwpfqo2.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-21T12:35:10.428 INFO:journalctl@ceph.mon.b.smithi104.stdout: RuntimeError: Failed command: systemctl restart ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek: Job for ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service failed because the control process exited with error code. 2024-09-21T12:35:10.428 INFO:journalctl@ceph.mon.b.smithi104.stdout: See "systemctl status ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service" and "journalctl -xeu ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service" for details. 2024-09-21T12:35:10.428 INFO:journalctl@ceph.mon.b.smithi104.stdout: Traceback (most recent call last): 2024-09-21T12:35:10.428 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1164, in _check_daemons 2024-09-21T12:35:10.429 INFO:journalctl@ceph.mon.b.smithi104.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-09-21T12:35:10.429 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2610, in _daemon_action 2024-09-21T12:35:10.429 INFO:journalctl@ceph.mon.b.smithi104.stdout: return self.wait_async( 2024-09-21T12:35:10.429 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 878, in wait_async 2024-09-21T12:35:10.429 INFO:journalctl@ceph.mon.b.smithi104.stdout: return self.event_loop.get_result(coro, timeout) 2024-09-21T12:35:10.429 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-09-21T12:35:10.429 INFO:journalctl@ceph.mon.b.smithi104.stdout: return future.result(timeout) 2024-09-21T12:35:10.429 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/lib64/python3.9/concurrent/futures/_base.py", line 446, in result 2024-09-21T12:35:10.429 INFO:journalctl@ceph.mon.b.smithi104.stdout: return self.__get_result() 2024-09-21T12:35:10.429 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/lib64/python3.9/concurrent/futures/_base.py", line 391, in __get_result 2024-09-21T12:35:10.429 INFO:journalctl@ceph.mon.b.smithi104.stdout: raise self._exception 2024-09-21T12:35:10.429 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1400, in _create_daemon 2024-09-21T12:35:10.429 INFO:journalctl@ceph.mon.b.smithi104.stdout: out, err, code = await self._run_cephadm( 2024-09-21T12:35:10.429 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1743, in _run_cephadm 2024-09-21T12:35:10.430 INFO:journalctl@ceph.mon.b.smithi104.stdout: raise OrchestratorError( 2024-09-21T12:35:10.430 INFO:journalctl@ceph.mon.b.smithi104.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi079.vlhrek ... 2024-09-21T12:35:10.430 INFO:journalctl@ceph.mon.b.smithi104.stdout: Creating ceph-iscsi config... 2024-09-21T12:35:10.430 INFO:journalctl@ceph.mon.b.smithi104.stdout: Write file: /var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/iscsi.foo.smithi079.vlhrek/iscsi-gateway.cfg 2024-09-21T12:35:10.430 INFO:journalctl@ceph.mon.b.smithi104.stdout: Write file: /var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/iscsi.foo.smithi079.vlhrek/tcmu-runner-entrypoint.sh 2024-09-21T12:35:10.430 INFO:journalctl@ceph.mon.b.smithi104.stdout: Non-zero exit code 1 from systemctl restart ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek 2024-09-21T12:35:10.430 INFO:journalctl@ceph.mon.b.smithi104.stdout: systemctl: stderr Job for ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service failed because the control process exited with error code. 2024-09-21T12:35:10.430 INFO:journalctl@ceph.mon.b.smithi104.stdout: systemctl: stderr See "systemctl status ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service" and "journalctl -xeu ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service" for details. 2024-09-21T12:35:10.430 INFO:journalctl@ceph.mon.b.smithi104.stdout: Traceback (most recent call last): 2024-09-21T12:35:10.430 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-21T12:35:10.430 INFO:journalctl@ceph.mon.b.smithi104.stdout: return _run_code(code, main_globals, None, 2024-09-21T12:35:10.430 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-21T12:35:10.430 INFO:journalctl@ceph.mon.b.smithi104.stdout: exec(code, run_globals) 2024-09-21T12:35:10.430 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 5628, in 2024-09-21T12:35:10.430 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 5616, in main 2024-09-21T12:35:10.431 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 3090, in command_deploy_from 2024-09-21T12:35:10.431 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 3125, in _common_deploy 2024-09-21T12:35:10.431 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 3145, in _deploy_daemon_container 2024-09-21T12:35:10.431 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 1147, in deploy_daemon 2024-09-21T12:35:10.431 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/tmp/tmptpwpfqo2.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-21T12:35:10.431 INFO:journalctl@ceph.mon.b.smithi104.stdout: RuntimeError: Failed command: systemctl restart ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek: Job for ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service failed because the control process exited with error code. 2024-09-21T12:35:10.431 INFO:journalctl@ceph.mon.b.smithi104.stdout: See "systemctl status ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service" and "journalctl -xeu ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service" for details. 2024-09-21T12:35:10.431 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:10 smithi104 ceph-mon[25639]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-21T12:35:10.431 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:10 smithi104 ceph-mon[25639]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 821 B/s rd, 1 op/s 2024-09-21T12:35:10.431 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:10 smithi104 ceph-mon[25639]: Reconfiguring daemon prometheus.a on smithi104 2024-09-21T12:35:12.142 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:12 smithi079 ceph-mon[23699]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 596 B/s rd, 0 op/s 2024-09-21T12:35:12.143 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:12 smithi079 ceph-mon[23699]: from='mgr.25054 ' entity='mgr.x' 2024-09-21T12:35:12.143 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:12 smithi079 ceph-mon[23699]: from='mgr.25054 ' entity='mgr.x' 2024-09-21T12:35:12.143 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:12 smithi079 ceph-mon[23699]: from='mgr.25054 ' entity='mgr.x' 2024-09-21T12:35:12.143 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:12 smithi079 ceph-mon[30023]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 596 B/s rd, 0 op/s 2024-09-21T12:35:12.143 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:12 smithi079 ceph-mon[30023]: from='mgr.25054 ' entity='mgr.x' 2024-09-21T12:35:12.143 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:12 smithi079 ceph-mon[30023]: from='mgr.25054 ' entity='mgr.x' 2024-09-21T12:35:12.143 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:12 smithi079 ceph-mon[30023]: from='mgr.25054 ' entity='mgr.x' 2024-09-21T12:35:12.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:12 smithi104 ceph-mon[25639]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 596 B/s rd, 0 op/s 2024-09-21T12:35:12.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:12 smithi104 ceph-mon[25639]: from='mgr.25054 ' entity='mgr.x' 2024-09-21T12:35:12.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:12 smithi104 ceph-mon[25639]: from='mgr.25054 ' entity='mgr.x' 2024-09-21T12:35:12.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:12 smithi104 ceph-mon[25639]: from='mgr.25054 ' entity='mgr.x' 2024-09-21T12:35:13.112 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:13 smithi079 ceph-mon[23699]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 676 B/s rd, 0 op/s 2024-09-21T12:35:13.112 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:13 smithi079 ceph-mon[23699]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 830 B/s rd, 1 op/s 2024-09-21T12:35:13.112 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:13 smithi079 ceph-mon[30023]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 676 B/s rd, 0 op/s 2024-09-21T12:35:13.112 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:13 smithi079 ceph-mon[30023]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 830 B/s rd, 1 op/s 2024-09-21T12:35:13.395 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:35:13 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[70883]: ::ffff:172.21.15.104 - - [21/Sep/2024:12:35:13] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-21T12:35:13.426 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:13 smithi104 ceph-mon[25639]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 676 B/s rd, 0 op/s 2024-09-21T12:35:13.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:13 smithi104 ceph-mon[25639]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 830 B/s rd, 1 op/s 2024-09-21T12:35:15.176 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:14 smithi104 ceph-mon[25639]: from='mgr.25054 ' entity='mgr.x' 2024-09-21T12:35:15.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:14 smithi104 ceph-mon[25639]: from='mgr.25054 ' entity='mgr.x' 2024-09-21T12:35:15.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:14 smithi104 ceph-mon[25639]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 705 B/s rd, 0 op/s 2024-09-21T12:35:15.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:14 smithi104 ceph-mon[25639]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 705 B/s rd, 0 op/s 2024-09-21T12:35:15.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:14 smithi104 ceph-mon[25639]: from='mgr.25054 ' entity='mgr.x' 2024-09-21T12:35:15.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:14 smithi104 ceph-mon[25639]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T12:35:15.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:14 smithi079 ceph-mon[23699]: from='mgr.25054 ' entity='mgr.x' 2024-09-21T12:35:15.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:14 smithi079 ceph-mon[23699]: from='mgr.25054 ' entity='mgr.x' 2024-09-21T12:35:15.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:14 smithi079 ceph-mon[23699]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 705 B/s rd, 0 op/s 2024-09-21T12:35:15.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:14 smithi079 ceph-mon[23699]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 705 B/s rd, 0 op/s 2024-09-21T12:35:15.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:14 smithi079 ceph-mon[23699]: from='mgr.25054 ' entity='mgr.x' 2024-09-21T12:35:15.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:14 smithi079 ceph-mon[23699]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T12:35:15.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:14 smithi079 ceph-mon[30023]: from='mgr.25054 ' entity='mgr.x' 2024-09-21T12:35:15.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:14 smithi079 ceph-mon[30023]: from='mgr.25054 ' entity='mgr.x' 2024-09-21T12:35:15.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:14 smithi079 ceph-mon[30023]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 705 B/s rd, 0 op/s 2024-09-21T12:35:15.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:14 smithi079 ceph-mon[30023]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 705 B/s rd, 0 op/s 2024-09-21T12:35:15.193 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:14 smithi079 ceph-mon[30023]: from='mgr.25054 ' entity='mgr.x' 2024-09-21T12:35:15.193 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:14 smithi079 ceph-mon[30023]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T12:35:16.868 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:16 smithi104 ceph-mon[25639]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 255 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:35:17.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:16 smithi079 ceph-mon[23699]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 255 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:35:17.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:16 smithi079 ceph-mon[30023]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 255 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:35:18.977 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:18 smithi104 ceph-mon[25639]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 255 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:35:19.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:18 smithi079 ceph-mon[30023]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 255 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:35:19.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:18 smithi079 ceph-mon[23699]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 255 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:35:19.372 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:35:19 smithi104 systemd[1]: Stopping Ceph prometheus.a for 33168022-7814-11ef-baf6-efdc52797490... 2024-09-21T12:35:19.641 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:35:19 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[61550]: ts=2024-09-21T12:35:19.371Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-09-21T12:35:19.641 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:35:19 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[61550]: ts=2024-09-21T12:35:19.371Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-09-21T12:35:19.641 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:35:19 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[61550]: ts=2024-09-21T12:35:19.371Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-09-21T12:35:19.641 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:35:19 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[61550]: ts=2024-09-21T12:35:19.371Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-09-21T12:35:19.642 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:35:19 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[61550]: ts=2024-09-21T12:35:19.371Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-09-21T12:35:19.642 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:35:19 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[61550]: ts=2024-09-21T12:35:19.371Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-09-21T12:35:19.642 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:35:19 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[61550]: ts=2024-09-21T12:35:19.371Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-09-21T12:35:19.642 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:35:19 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[61550]: ts=2024-09-21T12:35:19.371Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-09-21T12:35:19.642 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:35:19 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[61550]: ts=2024-09-21T12:35:19.371Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-09-21T12:35:19.642 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:35:19 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[61550]: ts=2024-09-21T12:35:19.371Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-09-21T12:35:19.642 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:35:19 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[61550]: ts=2024-09-21T12:35:19.371Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-09-21T12:35:19.642 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:35:19 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[61550]: ts=2024-09-21T12:35:19.371Z caller=main.go:1147 level=info msg="See you next time!" 2024-09-21T12:35:19.642 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:35:19 smithi104 podman[63454]: 2024-09-21 12:35:19.381374927 +0000 UTC m=+0.114479286 container died 724fc2fdc4783d91e1bcb13638f01f99aa4a02d728af22c7120b25bab269f5f9 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T12:35:19.926 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:35:19 smithi104 podman[63454]: 2024-09-21 12:35:19.748018453 +0000 UTC m=+0.481122807 container cleanup 724fc2fdc4783d91e1bcb13638f01f99aa4a02d728af22c7120b25bab269f5f9 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T12:35:19.927 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:35:19 smithi104 bash[63454]: ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a 2024-09-21T12:35:20.304 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:35:20 smithi104 podman[63466]: 2024-09-21 12:35:20.042703334 +0000 UTC m=+0.657687761 container remove 724fc2fdc4783d91e1bcb13638f01f99aa4a02d728af22c7120b25bab269f5f9 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T12:35:21.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:20 smithi104 ceph-mon[25639]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 255 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:35:21.177 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:35:20 smithi104 systemd[1]: ceph-33168022-7814-11ef-baf6-efdc52797490@prometheus.a.service: Deactivated successfully. 2024-09-21T12:35:21.178 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:35:20 smithi104 systemd[1]: Stopped Ceph prometheus.a for 33168022-7814-11ef-baf6-efdc52797490. 2024-09-21T12:35:21.178 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:35:20 smithi104 systemd[1]: Starting Ceph prometheus.a for 33168022-7814-11ef-baf6-efdc52797490... 2024-09-21T12:35:21.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:20 smithi079 ceph-mon[23699]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 255 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:35:21.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:20 smithi079 ceph-mon[30023]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 255 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:35:21.451 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:35:21 smithi104 podman[63595]: 2024-09-21 12:35:21.33971076 +0000 UTC m=+0.016675309 image pull a07b618ecd1dce142bce4c52f0e80982eaf1f14265a2415c2d35978ccaa0a464 quay.io/prometheus/prometheus:v2.43.0 2024-09-21T12:35:21.451 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:35:21 smithi104 podman[63595]: 2024-09-21 12:35:21.450442642 +0000 UTC m=+0.127407222 container create 79f2e38dc5df4d3dc9939222b5389c29e22dd165c870e7e5dd1ed78d68acad76 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T12:35:21.707 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:35:21 smithi104 podman[63595]: 2024-09-21 12:35:21.677929346 +0000 UTC m=+0.354893892 container init 79f2e38dc5df4d3dc9939222b5389c29e22dd165c870e7e5dd1ed78d68acad76 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T12:35:21.707 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:35:21 smithi104 podman[63595]: 2024-09-21 12:35:21.681447933 +0000 UTC m=+0.358412477 container start 79f2e38dc5df4d3dc9939222b5389c29e22dd165c870e7e5dd1ed78d68acad76 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T12:35:21.972 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:21 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: [21/Sep/2024:12:35:21] ENGINE Bus STOPPING 2024-09-21T12:35:21.972 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:21 smithi104 ceph-mon[25639]: from='client.? 172.21.15.79:0/2139117245' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-21T12:35:21.972 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:35:21 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[63610]: ts=2024-09-21T12:35:21.705Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-09-21T12:35:21.972 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:35:21 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[63610]: ts=2024-09-21T12:35:21.705Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-09-21T12:35:21.973 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:35:21 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[63610]: ts=2024-09-21T12:35:21.705Z caller=main.go:570 level=info host_details="(Linux 5.14.0-508.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024 x86_64 smithi104 (none))" 2024-09-21T12:35:21.973 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:35:21 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[63610]: ts=2024-09-21T12:35:21.705Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-09-21T12:35:21.973 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:35:21 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[63610]: ts=2024-09-21T12:35:21.705Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-09-21T12:35:21.973 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:35:21 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[63610]: ts=2024-09-21T12:35:21.707Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-09-21T12:35:21.973 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:35:21 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[63610]: ts=2024-09-21T12:35:21.708Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-09-21T12:35:21.973 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:35:21 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[63610]: ts=2024-09-21T12:35:21.709Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-09-21T12:35:21.973 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:35:21 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[63610]: ts=2024-09-21T12:35:21.709Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-09-21T12:35:21.973 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:35:21 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[63610]: ts=2024-09-21T12:35:21.710Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-09-21T12:35:21.973 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:35:21 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[63610]: ts=2024-09-21T12:35:21.710Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.365µs 2024-09-21T12:35:21.973 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:35:21 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[63610]: ts=2024-09-21T12:35:21.710Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-09-21T12:35:21.973 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:35:21 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[63610]: ts=2024-09-21T12:35:21.712Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=4 2024-09-21T12:35:21.973 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:35:21 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[63610]: ts=2024-09-21T12:35:21.720Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=4 2024-09-21T12:35:21.973 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:35:21 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[63610]: ts=2024-09-21T12:35:21.722Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=2 maxSegment=4 2024-09-21T12:35:21.973 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:35:21 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[63610]: ts=2024-09-21T12:35:21.723Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=3 maxSegment=4 2024-09-21T12:35:21.974 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:35:21 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[63610]: ts=2024-09-21T12:35:21.723Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=4 maxSegment=4 2024-09-21T12:35:21.974 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:35:21 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[63610]: ts=2024-09-21T12:35:21.723Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=33.611µs wal_replay_duration=13.074079ms wbl_replay_duration=129ns total_replay_duration=13.130733ms 2024-09-21T12:35:21.974 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:35:21 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[63610]: ts=2024-09-21T12:35:21.724Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-09-21T12:35:21.974 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:35:21 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[63610]: ts=2024-09-21T12:35:21.724Z caller=main.go:1029 level=info msg="TSDB started" 2024-09-21T12:35:21.974 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:35:21 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[63610]: ts=2024-09-21T12:35:21.724Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-09-21T12:35:21.974 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:35:21 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[63610]: ts=2024-09-21T12:35:21.745Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=20.492969ms db_storage=844ns remote_storage=1.186µs web_handler=427ns query_engine=907ns scrape=467.905µs scrape_sd=130.904µs notify=25.785µs notify_sd=27.43µs rules=19.47165ms tracing=6.869µs 2024-09-21T12:35:21.974 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:35:21 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[63610]: ts=2024-09-21T12:35:21.745Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-09-21T12:35:21.974 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:35:21 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[63610]: ts=2024-09-21T12:35:21.745Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-09-21T12:35:21.974 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:35:21 smithi104 bash[63595]: 79f2e38dc5df4d3dc9939222b5389c29e22dd165c870e7e5dd1ed78d68acad76 2024-09-21T12:35:21.974 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:35:21 smithi104 systemd[1]: Started Ceph prometheus.a for 33168022-7814-11ef-baf6-efdc52797490. 2024-09-21T12:35:22.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:21 smithi079 ceph-mon[23699]: from='client.? 172.21.15.79:0/2139117245' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-21T12:35:22.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:21 smithi079 ceph-mon[30023]: from='client.? 172.21.15.79:0/2139117245' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-21T12:35:22.570 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:22 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: [21/Sep/2024:12:35:22] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-21T12:35:22.570 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:22 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: [21/Sep/2024:12:35:22] ENGINE Bus STOPPED 2024-09-21T12:35:22.570 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:22 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: [21/Sep/2024:12:35:22] ENGINE Bus STARTING 2024-09-21T12:35:22.570 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:22 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: [21/Sep/2024:12:35:22] ENGINE Serving on http://:::9283 2024-09-21T12:35:22.570 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:22 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: [21/Sep/2024:12:35:22] ENGINE Bus STARTED 2024-09-21T12:35:22.903 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:22 smithi104 ceph-mon[25639]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 255 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:35:22.903 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:22 smithi104 ceph-mon[25639]: from='mgr.25054 ' entity='mgr.x' 2024-09-21T12:35:22.903 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:22 smithi104 ceph-mon[25639]: from='mgr.25054 ' entity='mgr.x' 2024-09-21T12:35:22.903 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:22 smithi104 ceph-mon[25639]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-21T12:35:22.904 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:22 smithi104 ceph-mon[25639]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-21T12:35:22.904 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:22 smithi104 ceph-mon[25639]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi104:9095"}]: dispatch 2024-09-21T12:35:22.904 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:22 smithi104 ceph-mon[25639]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi104:9095"}]: dispatch 2024-09-21T12:35:22.904 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:22 smithi104 ceph-mon[25639]: from='mgr.25054 ' entity='mgr.x' 2024-09-21T12:35:22.904 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:22 smithi104 ceph-mon[25639]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:35:22.904 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:22 smithi104 ceph-mon[25639]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T12:35:23.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:22 smithi079 ceph-mon[23699]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 255 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:35:23.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:22 smithi079 ceph-mon[23699]: from='mgr.25054 ' entity='mgr.x' 2024-09-21T12:35:23.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:22 smithi079 ceph-mon[23699]: from='mgr.25054 ' entity='mgr.x' 2024-09-21T12:35:23.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:22 smithi079 ceph-mon[23699]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-21T12:35:23.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:22 smithi079 ceph-mon[23699]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-21T12:35:23.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:22 smithi079 ceph-mon[23699]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi104:9095"}]: dispatch 2024-09-21T12:35:23.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:22 smithi079 ceph-mon[23699]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi104:9095"}]: dispatch 2024-09-21T12:35:23.193 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:22 smithi079 ceph-mon[23699]: from='mgr.25054 ' entity='mgr.x' 2024-09-21T12:35:23.193 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:22 smithi079 ceph-mon[23699]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:35:23.193 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:22 smithi079 ceph-mon[23699]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T12:35:23.193 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:22 smithi079 ceph-mon[30023]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 255 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:35:23.193 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:22 smithi079 ceph-mon[30023]: from='mgr.25054 ' entity='mgr.x' 2024-09-21T12:35:23.193 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:22 smithi079 ceph-mon[30023]: from='mgr.25054 ' entity='mgr.x' 2024-09-21T12:35:23.193 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:22 smithi079 ceph-mon[30023]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-21T12:35:23.193 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:22 smithi079 ceph-mon[30023]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-21T12:35:23.193 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:22 smithi079 ceph-mon[30023]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi104:9095"}]: dispatch 2024-09-21T12:35:23.193 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:22 smithi079 ceph-mon[30023]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi104:9095"}]: dispatch 2024-09-21T12:35:23.193 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:22 smithi079 ceph-mon[30023]: from='mgr.25054 ' entity='mgr.x' 2024-09-21T12:35:23.193 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:22 smithi079 ceph-mon[30023]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:35:23.194 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:22 smithi079 ceph-mon[30023]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T12:35:23.897 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:23 smithi079 ceph-mon[30023]: from='mon.? -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T12:35:23.897 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:23 smithi079 ceph-mon[30023]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-21T12:35:23.897 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:23 smithi079 ceph-mon[30023]: from='mgr.25054 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-21T12:35:23.897 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:23 smithi079 ceph-mon[30023]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-21T12:35:23.897 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:23 smithi079 ceph-mon[30023]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:35:23.897 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:23 smithi079 ceph-mon[23699]: from='mon.? -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T12:35:23.897 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:23 smithi079 ceph-mon[23699]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-21T12:35:23.897 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:23 smithi079 ceph-mon[23699]: from='mgr.25054 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-21T12:35:23.898 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:23 smithi079 ceph-mon[23699]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-21T12:35:23.898 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:23 smithi079 ceph-mon[23699]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:35:24.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:23 smithi104 ceph-mon[25639]: from='mon.? -' entity='mon.' cmd=[{"prefix": "orch prometheus get-credentials"}]: dispatch 2024-09-21T12:35:24.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:23 smithi104 ceph-mon[25639]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-21T12:35:24.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:23 smithi104 ceph-mon[25639]: from='mgr.25054 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-21T12:35:24.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:23 smithi104 ceph-mon[25639]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-21T12:35:24.080 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:23 smithi104 ceph-mon[25639]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:35:24.926 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:24 smithi104 ceph-mon[25639]: Upgrade: Updating mgr.y 2024-09-21T12:35:24.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:24 smithi104 ceph-mon[25639]: Deploying daemon mgr.y on smithi079 2024-09-21T12:35:24.927 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:24 smithi104 ceph-mon[25639]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 255 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:35:25.014 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:24 smithi079 ceph-mon[23699]: Upgrade: Updating mgr.y 2024-09-21T12:35:25.014 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:24 smithi079 ceph-mon[23699]: Deploying daemon mgr.y on smithi079 2024-09-21T12:35:25.014 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:24 smithi079 ceph-mon[23699]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 255 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:35:25.014 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:24 smithi079 ceph-mon[30023]: Upgrade: Updating mgr.y 2024-09-21T12:35:25.014 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:24 smithi079 ceph-mon[30023]: Deploying daemon mgr.y on smithi079 2024-09-21T12:35:25.014 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:24 smithi079 ceph-mon[30023]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 255 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:35:25.435 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:35:25 smithi079 systemd[1]: Stopping Ceph mgr.y for 33168022-7814-11ef-baf6-efdc52797490... 2024-09-21T12:35:25.457 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 33168022-7814-11ef-baf6-efdc52797490 -e sha1=441784946648ff53dd310293f9dd752afeeea72a -- bash -c 'ceph orch ps' 2024-09-21T12:35:25.997 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:35:25 smithi079 podman[76447]: 2024-09-21 12:35:25.718767751 +0000 UTC m=+0.120656149 container died b1744e3101ac1b54b1b4340f1666ce2a050e7a87cf0f3c5958dcd304e082ed63 (image=quay.ceph.io/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-guits-main-2024-09-21-0842-4417849, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-09-21T12:35:26.161 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:35:26.281 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:35:26 smithi079 podman[76447]: 2024-09-21 12:35:26.146186476 +0000 UTC m=+0.548074861 container cleanup b1744e3101ac1b54b1b4340f1666ce2a050e7a87cf0f3c5958dcd304e082ed63 (image=quay.ceph.io/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, ceph=True, GIT_CLEAN=True, RELEASE=wip-guits-main-2024-09-21-0842-4417849, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T12:35:26.282 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:35:26 smithi079 bash[76447]: ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y 2024-09-21T12:35:26.822 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:35:26 smithi079 podman[76508]: 2024-09-21 12:35:26.557942793 +0000 UTC m=+0.390921498 container died a7f0b213d06b41afbf3af4e75e4de0ad119c9691f0f10be847f9ddf56b198d85 (image=quay.ceph.io/ceph-ci/ceph:reef, name=intelligent_saha, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, RELEASE=reef-6a1bdf8, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-09-21T12:35:27.076 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:26 smithi104 ceph-mon[25639]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-09-21T12:35:27.078 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:26 smithi079 ceph-mon[23699]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-09-21T12:35:27.078 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:26 smithi079 ceph-mon[30023]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-09-21T12:35:27.381 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:35:27 smithi079 bash[76579]: ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y 2024-09-21T12:35:28.110 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:35:27 smithi079 podman[76491]: 2024-09-21 12:35:27.808620576 +0000 UTC m=+2.085443158 container remove b1744e3101ac1b54b1b4340f1666ce2a050e7a87cf0f3c5958dcd304e082ed63 (image=quay.ceph.io/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-guits-main-2024-09-21-0842-4417849, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731) 2024-09-21T12:35:28.110 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:35:27 smithi079 systemd[1]: ceph-33168022-7814-11ef-baf6-efdc52797490@mgr.y.service: Main process exited, code=exited, status=143/n/a 2024-09-21T12:35:28.111 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:35:27 smithi079 systemd[1]: ceph-33168022-7814-11ef-baf6-efdc52797490@mgr.y.service: Failed with result 'exit-code'. 2024-09-21T12:35:28.111 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:35:27 smithi079 systemd[1]: Stopped Ceph mgr.y for 33168022-7814-11ef-baf6-efdc52797490. 2024-09-21T12:35:28.111 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:35:27 smithi079 systemd[1]: ceph-33168022-7814-11ef-baf6-efdc52797490@mgr.y.service: Consumed 21.351s CPU time. 2024-09-21T12:35:28.111 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:35:28 smithi079 systemd[1]: Starting Ceph mgr.y for 33168022-7814-11ef-baf6-efdc52797490... 2024-09-21T12:35:28.893 INFO:teuthology.orchestra.run.smithi079.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-21T12:35:28.893 INFO:teuthology.orchestra.run.smithi079.stdout:agent.smithi079 smithi079 running 15s ago 11m - - 2024-09-21T12:35:28.893 INFO:teuthology.orchestra.run.smithi079.stdout:agent.smithi104 smithi104 running 31s ago 10m - - 2024-09-21T12:35:28.893 INFO:teuthology.orchestra.run.smithi079.stdout:alertmanager.a smithi079 *:9093,9094 running (5m) 15s ago 6m 26.9M - 0.25.0 c8568f914cd2 1451b4f42e50 2024-09-21T12:35:28.893 INFO:teuthology.orchestra.run.smithi079.stdout:grafana.a smithi104 *:3000 running (6m) 31s ago 6m 86.6M - 9.4.7 954c08fa6188 c85dd02f6519 2024-09-21T12:35:28.893 INFO:teuthology.orchestra.run.smithi079.stdout:iscsi.foo.smithi079.vlhrek smithi079 *:5000 unknown 15s ago 4m - - 2024-09-21T12:35:28.893 INFO:teuthology.orchestra.run.smithi079.stdout:mgr.x smithi104 *:8443,9283,8765 running (87s) 31s ago 10m 451M - 19.3.0-5104-g44178494 b415816db8b4 9038010f43b8 2024-09-21T12:35:28.893 INFO:teuthology.orchestra.run.smithi079.stdout:mgr.y smithi079 *:8443,9283,8765 running (48s) 15s ago 12m 451M - 19.3.0-5104-g44178494 b415816db8b4 b1744e3101ac 2024-09-21T12:35:28.893 INFO:teuthology.orchestra.run.smithi079.stdout:mon.a smithi079 running (12m) 15s ago 12m 59.2M 2048M 18.2.4-871-g6a1bdf85 0adab276c97e 794d66ac29c5 2024-09-21T12:35:28.893 INFO:teuthology.orchestra.run.smithi079.stdout:mon.b smithi104 running (10m) 31s ago 10m 46.6M 2048M 18.2.4-871-g6a1bdf85 0adab276c97e 460a6e8db654 2024-09-21T12:35:28.894 INFO:teuthology.orchestra.run.smithi079.stdout:mon.c smithi079 running (10m) 15s ago 10m 45.8M 2048M 18.2.4-871-g6a1bdf85 0adab276c97e 4b837a3d8ec3 2024-09-21T12:35:28.894 INFO:teuthology.orchestra.run.smithi079.stdout:node-exporter.a smithi079 *:9100 running (6m) 15s ago 6m 15.7M - 1.5.0 0da6a335fe13 f7bef67502dc 2024-09-21T12:35:28.894 INFO:teuthology.orchestra.run.smithi079.stdout:node-exporter.b smithi104 *:9100 running (6m) 31s ago 6m 16.1M - 1.5.0 0da6a335fe13 3701f535d078 2024-09-21T12:35:28.894 INFO:teuthology.orchestra.run.smithi079.stdout:osd.0 smithi079 running (10m) 15s ago 10m 60.4M 4096M 18.2.4-871-g6a1bdf85 0adab276c97e 46b0cbf47ea9 2024-09-21T12:35:28.894 INFO:teuthology.orchestra.run.smithi079.stdout:osd.1 smithi079 running (9m) 15s ago 9m 53.8M 4096M 18.2.4-871-g6a1bdf85 0adab276c97e 6f56f34ef26f 2024-09-21T12:35:28.894 INFO:teuthology.orchestra.run.smithi079.stdout:osd.2 smithi079 running (9m) 15s ago 9m 57.1M 4096M 18.2.4-871-g6a1bdf85 0adab276c97e 9a4767c65578 2024-09-21T12:35:28.894 INFO:teuthology.orchestra.run.smithi079.stdout:osd.3 smithi079 running (8m) 15s ago 8m 51.2M 4096M 18.2.4-871-g6a1bdf85 0adab276c97e 1e7af3834ad3 2024-09-21T12:35:28.894 INFO:teuthology.orchestra.run.smithi079.stdout:osd.4 smithi104 running (8m) 31s ago 8m 62.8M 4096M 18.2.4-871-g6a1bdf85 0adab276c97e 2b027321f961 2024-09-21T12:35:28.894 INFO:teuthology.orchestra.run.smithi079.stdout:osd.5 smithi104 running (8m) 31s ago 8m 58.3M 4096M 18.2.4-871-g6a1bdf85 0adab276c97e 5e738fa75e97 2024-09-21T12:35:28.894 INFO:teuthology.orchestra.run.smithi079.stdout:osd.6 smithi104 running (7m) 31s ago 7m 57.8M 4096M 18.2.4-871-g6a1bdf85 0adab276c97e fd2852c1818e 2024-09-21T12:35:28.894 INFO:teuthology.orchestra.run.smithi079.stdout:osd.7 smithi104 running (7m) 31s ago 7m 52.0M 4096M 18.2.4-871-g6a1bdf85 0adab276c97e d2b958c30fc4 2024-09-21T12:35:28.894 INFO:teuthology.orchestra.run.smithi079.stdout:prometheus.a smithi104 *:9095 starting - - - - 2024-09-21T12:35:28.894 INFO:teuthology.orchestra.run.smithi079.stdout:rgw.foo.smithi079.dfrddk smithi079 *:8000 running (5m) 15s ago 5m 78.3M - 18.2.4-871-g6a1bdf85 0adab276c97e 16a684a83189 2024-09-21T12:35:28.894 INFO:teuthology.orchestra.run.smithi079.stdout:rgw.foo.smithi104.qonbxe smithi104 *:8000 running (5m) 31s ago 5m 78.5M - 18.2.4-871-g6a1bdf85 0adab276c97e c9f39a22a235 2024-09-21T12:35:28.895 INFO:teuthology.orchestra.run.smithi079.stdout:rgw.smpl.smithi079.tydkfh smithi079 *:80 running (4m) 15s ago 4m 78.7M - 18.2.4-871-g6a1bdf85 0adab276c97e f59806e2a744 2024-09-21T12:35:28.895 INFO:teuthology.orchestra.run.smithi079.stdout:rgw.smpl.smithi104.riklnj smithi104 *:80 running (4m) 31s ago 4m 78.1M - 18.2.4-871-g6a1bdf85 0adab276c97e 47c316d12d6a 2024-09-21T12:35:29.033 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:28 smithi079 ceph-mon[23699]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-09-21T12:35:29.033 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:35:28 smithi079 podman[76787]: 2024-09-21 12:35:28.824069699 +0000 UTC m=+0.016489910 image pull b415816db8b41f544cdfdc71d193ff8b1ee008bf7b0ec4f64da8eb8f71b62e85 quay.ceph.io/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5 2024-09-21T12:35:29.034 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:35:28 smithi079 podman[76787]: 2024-09-21 12:35:28.925154775 +0000 UTC m=+0.117574971 container create b129d767810ff43a3a72917eb494d55dcf5aa1e366fc560c493701eb5423f133 (image=quay.ceph.io/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y, ceph=True, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, RELEASE=wip-guits-main-2024-09-21-0842-4417849, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731) 2024-09-21T12:35:29.034 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:28 smithi079 ceph-mon[30023]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-09-21T12:35:29.176 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:28 smithi104 ceph-mon[25639]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-09-21T12:35:29.930 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 33168022-7814-11ef-baf6-efdc52797490 -e sha1=441784946648ff53dd310293f9dd752afeeea72a -- bash -c 'ceph versions' 2024-09-21T12:35:30.109 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:29 smithi079 ceph-mon[23699]: from='client.15507 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:35:30.109 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:29 smithi079 ceph-mon[23699]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T12:35:30.109 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:29 smithi079 ceph-mon[30023]: from='client.15507 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:35:30.109 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:29 smithi079 ceph-mon[30023]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T12:35:30.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:29 smithi104 ceph-mon[25639]: from='client.15507 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:35:30.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:29 smithi104 ceph-mon[25639]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T12:35:30.349 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:35:30.441 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:35:30 smithi079 podman[76787]: 2024-09-21 12:35:30.108361322 +0000 UTC m=+1.300781518 container init b129d767810ff43a3a72917eb494d55dcf5aa1e366fc560c493701eb5423f133 (image=quay.ceph.io/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, RELEASE=wip-guits-main-2024-09-21-0842-4417849, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD) 2024-09-21T12:35:30.441 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:35:30 smithi079 podman[76787]: 2024-09-21 12:35:30.111961474 +0000 UTC m=+1.304381668 container start b129d767810ff43a3a72917eb494d55dcf5aa1e366fc560c493701eb5423f133 (image=quay.ceph.io/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, RELEASE=wip-guits-main-2024-09-21-0842-4417849, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2) 2024-09-21T12:35:30.441 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:35:30 smithi079 ceph-mgr[76908]: -- 172.21.15.79:0/727929077 <== mon.2 v2:172.21.15.79:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x562547cd10e0 con 0x562547caf400 2024-09-21T12:35:30.441 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:35:30 smithi079 bash[76787]: b129d767810ff43a3a72917eb494d55dcf5aa1e366fc560c493701eb5423f133 2024-09-21T12:35:30.441 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:35:30 smithi079 systemd[1]: Started Ceph mgr.y for 33168022-7814-11ef-baf6-efdc52797490. 2024-09-21T12:35:31.059 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:30 smithi079 ceph-mon[30023]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T12:35:31.059 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:30 smithi079 ceph-mon[30023]: from='mgr.25054 ' entity='mgr.x' 2024-09-21T12:35:31.059 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:30 smithi079 ceph-mon[30023]: from='mgr.25054 ' entity='mgr.x' 2024-09-21T12:35:31.059 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:30 smithi079 ceph-mon[30023]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:35:31.059 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:30 smithi079 ceph-mon[30023]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:35:31.059 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:30 smithi079 ceph-mon[30023]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:35:31.059 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:30 smithi079 ceph-mon[30023]: from='mgr.25054 ' entity='mgr.x' 2024-09-21T12:35:31.059 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:30 smithi079 ceph-mon[30023]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi079.vlhrek", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-21T12:35:31.059 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:30 smithi079 ceph-mon[30023]: from='mgr.25054 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi079.vlhrek", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-21T12:35:31.060 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:30 smithi079 ceph-mon[30023]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:35:31.060 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:30 smithi079 ceph-mon[23699]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T12:35:31.060 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:30 smithi079 ceph-mon[23699]: from='mgr.25054 ' entity='mgr.x' 2024-09-21T12:35:31.060 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:30 smithi079 ceph-mon[23699]: from='mgr.25054 ' entity='mgr.x' 2024-09-21T12:35:31.060 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:30 smithi079 ceph-mon[23699]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:35:31.060 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:30 smithi079 ceph-mon[23699]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:35:31.060 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:30 smithi079 ceph-mon[23699]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:35:31.060 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:30 smithi079 ceph-mon[23699]: from='mgr.25054 ' entity='mgr.x' 2024-09-21T12:35:31.060 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:30 smithi079 ceph-mon[23699]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi079.vlhrek", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-21T12:35:31.060 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:30 smithi079 ceph-mon[23699]: from='mgr.25054 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi079.vlhrek", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-21T12:35:31.060 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:30 smithi079 ceph-mon[23699]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:35:31.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:30 smithi104 ceph-mon[25639]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T12:35:31.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:30 smithi104 ceph-mon[25639]: from='mgr.25054 ' entity='mgr.x' 2024-09-21T12:35:31.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:30 smithi104 ceph-mon[25639]: from='mgr.25054 ' entity='mgr.x' 2024-09-21T12:35:31.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:30 smithi104 ceph-mon[25639]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:35:31.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:30 smithi104 ceph-mon[25639]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:35:31.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:30 smithi104 ceph-mon[25639]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:35:31.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:30 smithi104 ceph-mon[25639]: from='mgr.25054 ' entity='mgr.x' 2024-09-21T12:35:31.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:30 smithi104 ceph-mon[25639]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi079.vlhrek", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-21T12:35:31.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:30 smithi104 ceph-mon[25639]: from='mgr.25054 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi079.vlhrek", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-21T12:35:31.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:30 smithi104 ceph-mon[25639]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:35:31.941 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:31 smithi079 ceph-mon[23699]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 869 B/s rd, 0 op/s 2024-09-21T12:35:31.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:31 smithi079 ceph-mon[23699]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-21T12:35:31.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:31 smithi079 ceph-mon[23699]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-21T12:35:31.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:31 smithi079 ceph-mon[23699]: Reconfiguring iscsi.foo.smithi079.vlhrek (dependencies changed)... 2024-09-21T12:35:31.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:31 smithi079 ceph-mon[23699]: Reconfiguring daemon iscsi.foo.smithi079.vlhrek on smithi079 2024-09-21T12:35:31.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:31 smithi079 ceph-mon[23699]: from='client.25370 -' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:35:31.942 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:31 smithi079 ceph-mon[30023]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 869 B/s rd, 0 op/s 2024-09-21T12:35:31.942 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:31 smithi079 ceph-mon[30023]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-21T12:35:31.942 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:31 smithi079 ceph-mon[30023]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-21T12:35:31.942 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:31 smithi079 ceph-mon[30023]: Reconfiguring iscsi.foo.smithi079.vlhrek (dependencies changed)... 2024-09-21T12:35:31.943 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:31 smithi079 ceph-mon[30023]: Reconfiguring daemon iscsi.foo.smithi079.vlhrek on smithi079 2024-09-21T12:35:31.943 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:31 smithi079 ceph-mon[30023]: from='client.25370 -' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:35:32.176 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:31 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: ::ffff:172.21.15.104 - - [21/Sep/2024:12:35:31] "GET /metrics HTTP/1.1" 200 38580 "" "Prometheus/2.43.0" 2024-09-21T12:35:32.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:31 smithi104 ceph-mon[25639]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 869 B/s rd, 0 op/s 2024-09-21T12:35:32.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:31 smithi104 ceph-mon[25639]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-21T12:35:32.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:31 smithi104 ceph-mon[25639]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-21T12:35:32.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:31 smithi104 ceph-mon[25639]: Reconfiguring iscsi.foo.smithi079.vlhrek (dependencies changed)... 2024-09-21T12:35:32.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:31 smithi104 ceph-mon[25639]: Reconfiguring daemon iscsi.foo.smithi079.vlhrek on smithi079 2024-09-21T12:35:32.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:31 smithi104 ceph-mon[25639]: from='client.25370 -' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T12:35:34.063 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:33 smithi079 ceph-mon[23699]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-21T12:35:34.064 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:33 smithi079 ceph-mon[30023]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-21T12:35:34.064 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-21T12:35:34.064 INFO:teuthology.orchestra.run.smithi079.stdout: "mon": { 2024-09-21T12:35:34.064 INFO:teuthology.orchestra.run.smithi079.stdout: "ceph version 18.2.4-871-g6a1bdf85 (6a1bdf854bb5803efd58957907442aa01f41a8fc) reef (stable)": 3 2024-09-21T12:35:34.064 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-21T12:35:34.064 INFO:teuthology.orchestra.run.smithi079.stdout: "mgr": { 2024-09-21T12:35:34.064 INFO:teuthology.orchestra.run.smithi079.stdout: "ceph version 19.3.0-5104-g44178494 (441784946648ff53dd310293f9dd752afeeea72a) squid (dev)": 2 2024-09-21T12:35:34.064 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-21T12:35:34.064 INFO:teuthology.orchestra.run.smithi079.stdout: "osd": { 2024-09-21T12:35:34.064 INFO:teuthology.orchestra.run.smithi079.stdout: "ceph version 18.2.4-871-g6a1bdf85 (6a1bdf854bb5803efd58957907442aa01f41a8fc) reef (stable)": 8 2024-09-21T12:35:34.065 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-21T12:35:34.065 INFO:teuthology.orchestra.run.smithi079.stdout: "rgw": { 2024-09-21T12:35:34.065 INFO:teuthology.orchestra.run.smithi079.stdout: "ceph version 18.2.4-871-g6a1bdf85 (6a1bdf854bb5803efd58957907442aa01f41a8fc) reef (stable)": 4 2024-09-21T12:35:34.065 INFO:teuthology.orchestra.run.smithi079.stdout: }, 2024-09-21T12:35:34.065 INFO:teuthology.orchestra.run.smithi079.stdout: "overall": { 2024-09-21T12:35:34.065 INFO:teuthology.orchestra.run.smithi079.stdout: "ceph version 18.2.4-871-g6a1bdf85 (6a1bdf854bb5803efd58957907442aa01f41a8fc) reef (stable)": 15, 2024-09-21T12:35:34.065 INFO:teuthology.orchestra.run.smithi079.stdout: "ceph version 19.3.0-5104-g44178494 (441784946648ff53dd310293f9dd752afeeea72a) squid (dev)": 2 2024-09-21T12:35:34.065 INFO:teuthology.orchestra.run.smithi079.stdout: } 2024-09-21T12:35:34.065 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-21T12:35:34.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:33 smithi104 ceph-mon[25639]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-21T12:35:35.016 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 33168022-7814-11ef-baf6-efdc52797490 -e sha1=441784946648ff53dd310293f9dd752afeeea72a -- bash -c 'ceph orch upgrade status' 2024-09-21T12:35:35.176 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:34 smithi104 ceph-mon[25639]: from='mgr.25054 ' entity='mgr.x' 2024-09-21T12:35:35.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:34 smithi104 ceph-mon[25639]: from='mgr.25054 ' entity='mgr.x' 2024-09-21T12:35:35.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:34 smithi104 ceph-mon[25639]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 628 B/s rd, 0 op/s 2024-09-21T12:35:35.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:34 smithi104 ceph-mon[25639]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 833 B/s rd, 0 op/s 2024-09-21T12:35:35.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:34 smithi104 ceph-mon[25639]: from='mgr.25054 ' entity='mgr.x' 2024-09-21T12:35:35.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:34 smithi104 ceph-mon[25639]: from='client.? 172.21.15.79:0/4135232037' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T12:35:35.180 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:34 smithi079 ceph-mon[30023]: from='mgr.25054 ' entity='mgr.x' 2024-09-21T12:35:35.181 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:34 smithi079 ceph-mon[30023]: from='mgr.25054 ' entity='mgr.x' 2024-09-21T12:35:35.181 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:34 smithi079 ceph-mon[30023]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 628 B/s rd, 0 op/s 2024-09-21T12:35:35.181 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:34 smithi079 ceph-mon[30023]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 833 B/s rd, 0 op/s 2024-09-21T12:35:35.181 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:34 smithi079 ceph-mon[30023]: from='mgr.25054 ' entity='mgr.x' 2024-09-21T12:35:35.181 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:34 smithi079 ceph-mon[30023]: from='client.? 172.21.15.79:0/4135232037' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T12:35:35.181 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:34 smithi079 ceph-mon[23699]: from='mgr.25054 ' entity='mgr.x' 2024-09-21T12:35:35.181 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:34 smithi079 ceph-mon[23699]: from='mgr.25054 ' entity='mgr.x' 2024-09-21T12:35:35.181 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:34 smithi079 ceph-mon[23699]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 628 B/s rd, 0 op/s 2024-09-21T12:35:35.182 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:34 smithi079 ceph-mon[23699]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 833 B/s rd, 0 op/s 2024-09-21T12:35:35.182 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:34 smithi079 ceph-mon[23699]: from='mgr.25054 ' entity='mgr.x' 2024-09-21T12:35:35.182 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:34 smithi079 ceph-mon[23699]: from='client.? 172.21.15.79:0/4135232037' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T12:35:35.692 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:35:35 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[76858]: [21/Sep/2024:12:35:35] ENGINE Bus STARTING 2024-09-21T12:35:35.692 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:35:35 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[76858]: CherryPy Checker: 2024-09-21T12:35:35.692 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:35:35 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[76858]: The Application mounted at '' has an empty config. 2024-09-21T12:35:35.693 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:35:35 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[76858]: 2024-09-21T12:35:35.693 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:35:35 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[76858]: [21/Sep/2024:12:35:35] ENGINE Serving on http://:::9283 2024-09-21T12:35:35.693 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:35:35 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[76858]: [21/Sep/2024:12:35:35] ENGINE Bus STARTED 2024-09-21T12:35:36.093 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:35 smithi079 ceph-mon[23699]: Standby manager daemon y restarted 2024-09-21T12:35:36.093 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:35 smithi079 ceph-mon[23699]: Standby manager daemon y started 2024-09-21T12:35:36.093 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:35 smithi079 ceph-mon[23699]: from='mgr.? 172.21.15.79:0/3162128656' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-21T12:35:36.093 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:35 smithi079 ceph-mon[23699]: from='mgr.? 172.21.15.79:0/3162128656' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-21T12:35:36.094 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:35 smithi079 ceph-mon[23699]: from='mgr.? 172.21.15.79:0/3162128656' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-09-21T12:35:36.094 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:35 smithi079 ceph-mon[23699]: from='mgr.? 172.21.15.79:0/3162128656' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-21T12:35:36.094 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:35 smithi079 ceph-mon[30023]: Standby manager daemon y restarted 2024-09-21T12:35:36.094 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:35 smithi079 ceph-mon[30023]: Standby manager daemon y started 2024-09-21T12:35:36.094 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:35 smithi079 ceph-mon[30023]: from='mgr.? 172.21.15.79:0/3162128656' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-21T12:35:36.094 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:35 smithi079 ceph-mon[30023]: from='mgr.? 172.21.15.79:0/3162128656' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-21T12:35:36.094 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:35 smithi079 ceph-mon[30023]: from='mgr.? 172.21.15.79:0/3162128656' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-09-21T12:35:36.094 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:35 smithi079 ceph-mon[30023]: from='mgr.? 172.21.15.79:0/3162128656' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-21T12:35:36.166 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:35:36.176 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:35 smithi104 ceph-mon[25639]: Standby manager daemon y restarted 2024-09-21T12:35:36.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:35 smithi104 ceph-mon[25639]: Standby manager daemon y started 2024-09-21T12:35:36.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:35 smithi104 ceph-mon[25639]: from='mgr.? 172.21.15.79:0/3162128656' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-21T12:35:36.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:35 smithi104 ceph-mon[25639]: from='mgr.? 172.21.15.79:0/3162128656' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-21T12:35:36.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:35 smithi104 ceph-mon[25639]: from='mgr.? 172.21.15.79:0/3162128656' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-09-21T12:35:36.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:35 smithi104 ceph-mon[25639]: from='mgr.? 172.21.15.79:0/3162128656' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-21T12:35:37.175 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:36 smithi079 ceph-mon[30023]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 833 B/s rd, 0 op/s 2024-09-21T12:35:37.175 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:36 smithi079 ceph-mon[30023]: mgrmap e37: x(active, since 37s), standbys: y 2024-09-21T12:35:37.176 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:36 smithi079 ceph-mon[23699]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 833 B/s rd, 0 op/s 2024-09-21T12:35:37.176 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:36 smithi079 ceph-mon[23699]: mgrmap e37: x(active, since 37s), standbys: y 2024-09-21T12:35:37.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:36 smithi104 ceph-mon[25639]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 833 B/s rd, 0 op/s 2024-09-21T12:35:37.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:36 smithi104 ceph-mon[25639]: mgrmap e37: x(active, since 37s), standbys: y 2024-09-21T12:35:37.931 INFO:teuthology.orchestra.run.smithi079.stdout:{ 2024-09-21T12:35:37.931 INFO:teuthology.orchestra.run.smithi079.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5", 2024-09-21T12:35:37.931 INFO:teuthology.orchestra.run.smithi079.stdout: "in_progress": true, 2024-09-21T12:35:37.931 INFO:teuthology.orchestra.run.smithi079.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-21T12:35:37.931 INFO:teuthology.orchestra.run.smithi079.stdout: "services_complete": [ 2024-09-21T12:35:37.931 INFO:teuthology.orchestra.run.smithi079.stdout: "mgr" 2024-09-21T12:35:37.931 INFO:teuthology.orchestra.run.smithi079.stdout: ], 2024-09-21T12:35:37.931 INFO:teuthology.orchestra.run.smithi079.stdout: "progress": "2/23 daemons upgraded", 2024-09-21T12:35:37.931 INFO:teuthology.orchestra.run.smithi079.stdout: "message": "Currently upgrading mgr daemons", 2024-09-21T12:35:37.932 INFO:teuthology.orchestra.run.smithi079.stdout: "is_paused": false 2024-09-21T12:35:37.932 INFO:teuthology.orchestra.run.smithi079.stdout:} 2024-09-21T12:35:39.039 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:38 smithi079 ceph-mon[23699]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 678 B/s rd, 0 op/s 2024-09-21T12:35:39.039 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:38 smithi079 ceph-mon[23699]: from='client.25403 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:35:39.039 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:38 smithi079 ceph-mon[30023]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 678 B/s rd, 0 op/s 2024-09-21T12:35:39.039 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:38 smithi079 ceph-mon[30023]: from='client.25403 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:35:39.063 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 33168022-7814-11ef-baf6-efdc52797490 -e sha1=441784946648ff53dd310293f9dd752afeeea72a -- bash -c 'ceph health detail' 2024-09-21T12:35:39.176 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:38 smithi104 ceph-mon[25639]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 678 B/s rd, 0 op/s 2024-09-21T12:35:39.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:38 smithi104 ceph-mon[25639]: from='client.25403 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T12:35:40.141 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:35:40.941 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:40 smithi079 ceph-mon[23699]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 965 B/s rd, 1 op/s 2024-09-21T12:35:40.942 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:40 smithi079 ceph-mon[30023]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 965 B/s rd, 1 op/s 2024-09-21T12:35:41.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:40 smithi104 ceph-mon[25639]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 965 B/s rd, 1 op/s 2024-09-21T12:35:41.177 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:40 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: 2024-09-21T12:35:40.894+0000 7ff086688640 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi079.vlhrek ... 2024-09-21T12:35:41.177 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:40 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: Creating ceph-iscsi config... 2024-09-21T12:35:41.177 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:40 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: Write file: /var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/iscsi.foo.smithi079.vlhrek/iscsi-gateway.cfg 2024-09-21T12:35:41.177 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:40 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: Write file: /var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/iscsi.foo.smithi079.vlhrek/tcmu-runner-entrypoint.sh 2024-09-21T12:35:41.177 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:40 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: Non-zero exit code 1 from systemctl restart ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek 2024-09-21T12:35:41.177 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:40 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: systemctl: stderr Job for ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service failed because the control process exited with error code. 2024-09-21T12:35:41.177 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:40 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: systemctl: stderr See "systemctl status ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service" and "journalctl -xeu ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service" for details. 2024-09-21T12:35:41.178 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:40 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: Traceback (most recent call last): 2024-09-21T12:35:41.178 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:40 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-21T12:35:41.178 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:40 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: return _run_code(code, main_globals, None, 2024-09-21T12:35:41.178 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:40 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-21T12:35:41.178 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:40 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: exec(code, run_globals) 2024-09-21T12:35:41.178 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:40 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 5628, in 2024-09-21T12:35:41.178 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:40 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 5616, in main 2024-09-21T12:35:41.178 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:40 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 3090, in command_deploy_from 2024-09-21T12:35:41.178 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:40 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 3125, in _common_deploy 2024-09-21T12:35:41.178 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:40 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 3145, in _deploy_daemon_container 2024-09-21T12:35:41.178 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:40 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 1147, in deploy_daemon 2024-09-21T12:35:41.178 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:40 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: File "/tmp/tmptpwpfqo2.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-21T12:35:41.178 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:40 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: RuntimeError: Failed command: systemctl restart ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek: Job for ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service failed because the control process exited with error code. 2024-09-21T12:35:41.179 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:40 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: See "systemctl status ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service" and "journalctl -xeu ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service" for details. 2024-09-21T12:35:41.179 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:40 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: Traceback (most recent call last): 2024-09-21T12:35:41.179 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:40 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1164, in _check_daemons 2024-09-21T12:35:41.179 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:40 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: self.mgr._daemon_action(daemon_spec, action=action) 2024-09-21T12:35:41.179 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:40 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2610, in _daemon_action 2024-09-21T12:35:41.179 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:40 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: return self.wait_async( 2024-09-21T12:35:41.180 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:40 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: File "/usr/share/ceph/mgr/cephadm/module.py", line 878, in wait_async 2024-09-21T12:35:41.180 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:40 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: return self.event_loop.get_result(coro, timeout) 2024-09-21T12:35:41.180 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:40 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-09-21T12:35:41.180 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:40 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: return future.result(timeout) 2024-09-21T12:35:41.180 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:40 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: File "/lib64/python3.9/concurrent/futures/_base.py", line 446, in result 2024-09-21T12:35:41.180 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:40 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: return self.__get_result() 2024-09-21T12:35:41.180 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:40 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: File "/lib64/python3.9/concurrent/futures/_base.py", line 391, in __get_result 2024-09-21T12:35:41.180 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:40 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: raise self._exception 2024-09-21T12:35:41.180 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:40 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1400, in _create_daemon 2024-09-21T12:35:41.180 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:40 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: out, err, code = await self._run_cephadm( 2024-09-21T12:35:41.180 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:40 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1743, in _run_cephadm 2024-09-21T12:35:41.180 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:40 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: raise OrchestratorError( 2024-09-21T12:35:41.180 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:40 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi079.vlhrek ... 2024-09-21T12:35:41.180 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:40 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: Creating ceph-iscsi config... 2024-09-21T12:35:41.181 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:40 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: Write file: /var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/iscsi.foo.smithi079.vlhrek/iscsi-gateway.cfg 2024-09-21T12:35:41.181 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:40 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: Write file: /var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/iscsi.foo.smithi079.vlhrek/tcmu-runner-entrypoint.sh 2024-09-21T12:35:41.181 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:40 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: Non-zero exit code 1 from systemctl restart ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek 2024-09-21T12:35:41.181 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:40 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: systemctl: stderr Job for ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service failed because the control process exited with error code. 2024-09-21T12:35:41.181 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:40 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: systemctl: stderr See "systemctl status ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service" and "journalctl -xeu ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service" for details. 2024-09-21T12:35:41.181 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:40 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: Traceback (most recent call last): 2024-09-21T12:35:41.181 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:40 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-21T12:35:41.181 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:40 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: return _run_code(code, main_globals, None, 2024-09-21T12:35:41.181 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:40 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-21T12:35:41.181 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:40 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: exec(code, run_globals) 2024-09-21T12:35:41.181 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:40 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 5628, in 2024-09-21T12:35:41.181 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:40 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 5616, in main 2024-09-21T12:35:41.181 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:40 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 3090, in command_deploy_from 2024-09-21T12:35:41.182 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:40 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 3125, in _common_deploy 2024-09-21T12:35:41.182 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:40 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 3145, in _deploy_daemon_container 2024-09-21T12:35:41.182 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:40 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 1147, in deploy_daemon 2024-09-21T12:35:41.182 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:40 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: File "/tmp/tmptpwpfqo2.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-21T12:35:41.182 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:40 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: RuntimeError: Failed command: systemctl restart ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek: Job for ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service failed because the control process exited with error code. 2024-09-21T12:35:41.182 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:40 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: See "systemctl status ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service" and "journalctl -xeu ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service" for details. 2024-09-21T12:35:42.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:41 smithi104 ceph-mon[25639]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi079.vlhrek ... 2024-09-21T12:35:42.177 INFO:journalctl@ceph.mon.b.smithi104.stdout: Creating ceph-iscsi config... 2024-09-21T12:35:42.177 INFO:journalctl@ceph.mon.b.smithi104.stdout: Write file: /var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/iscsi.foo.smithi079.vlhrek/iscsi-gateway.cfg 2024-09-21T12:35:42.177 INFO:journalctl@ceph.mon.b.smithi104.stdout: Write file: /var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/iscsi.foo.smithi079.vlhrek/tcmu-runner-entrypoint.sh 2024-09-21T12:35:42.178 INFO:journalctl@ceph.mon.b.smithi104.stdout: Non-zero exit code 1 from systemctl restart ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek 2024-09-21T12:35:42.178 INFO:journalctl@ceph.mon.b.smithi104.stdout: systemctl: stderr Job for ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service failed because the control process exited with error code. 2024-09-21T12:35:42.178 INFO:journalctl@ceph.mon.b.smithi104.stdout: systemctl: stderr See "systemctl status ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service" and "journalctl -xeu ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service" for details. 2024-09-21T12:35:42.178 INFO:journalctl@ceph.mon.b.smithi104.stdout: Traceback (most recent call last): 2024-09-21T12:35:42.178 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-21T12:35:42.178 INFO:journalctl@ceph.mon.b.smithi104.stdout: return _run_code(code, main_globals, None, 2024-09-21T12:35:42.178 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-21T12:35:42.178 INFO:journalctl@ceph.mon.b.smithi104.stdout: exec(code, run_globals) 2024-09-21T12:35:42.179 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 5628, in 2024-09-21T12:35:42.179 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 5616, in main 2024-09-21T12:35:42.179 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 3090, in command_deploy_from 2024-09-21T12:35:42.179 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 3125, in _common_deploy 2024-09-21T12:35:42.179 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 3145, in _deploy_daemon_container 2024-09-21T12:35:42.179 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 1147, in deploy_daemon 2024-09-21T12:35:42.179 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/tmp/tmptpwpfqo2.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-21T12:35:42.179 INFO:journalctl@ceph.mon.b.smithi104.stdout: RuntimeError: Failed command: systemctl restart ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek: Job for ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service failed because the control process exited with error code. 2024-09-21T12:35:42.179 INFO:journalctl@ceph.mon.b.smithi104.stdout: See "systemctl status ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service" and "journalctl -xeu ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service" for details. 2024-09-21T12:35:42.179 INFO:journalctl@ceph.mon.b.smithi104.stdout: Traceback (most recent call last): 2024-09-21T12:35:42.180 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1164, in _check_daemons 2024-09-21T12:35:42.180 INFO:journalctl@ceph.mon.b.smithi104.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-09-21T12:35:42.180 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2610, in _daemon_action 2024-09-21T12:35:42.180 INFO:journalctl@ceph.mon.b.smithi104.stdout: return self.wait_async( 2024-09-21T12:35:42.180 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 878, in wait_async 2024-09-21T12:35:42.180 INFO:journalctl@ceph.mon.b.smithi104.stdout: return self.event_loop.get_result(coro, timeout) 2024-09-21T12:35:42.180 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-09-21T12:35:42.180 INFO:journalctl@ceph.mon.b.smithi104.stdout: return future.result(timeout) 2024-09-21T12:35:42.180 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/lib64/python3.9/concurrent/futures/_base.py", line 446, in result 2024-09-21T12:35:42.181 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:41 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: ::ffff:172.21.15.104 - - [21/Sep/2024:12:35:41] "GET /metrics HTTP/1.1" 200 38580 "" "Prometheus/2.43.0" 2024-09-21T12:35:42.181 INFO:journalctl@ceph.mon.b.smithi104.stdout: return self.__get_result() 2024-09-21T12:35:42.182 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/lib64/python3.9/concurrent/futures/_base.py", line 391, in __get_result 2024-09-21T12:35:42.182 INFO:journalctl@ceph.mon.b.smithi104.stdout: raise self._exception 2024-09-21T12:35:42.182 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1400, in _create_daemon 2024-09-21T12:35:42.182 INFO:journalctl@ceph.mon.b.smithi104.stdout: out, err, code = await self._run_cephadm( 2024-09-21T12:35:42.182 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1743, in _run_cephadm 2024-09-21T12:35:42.182 INFO:journalctl@ceph.mon.b.smithi104.stdout: raise OrchestratorError( 2024-09-21T12:35:42.182 INFO:journalctl@ceph.mon.b.smithi104.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi079.vlhrek ... 2024-09-21T12:35:42.183 INFO:journalctl@ceph.mon.b.smithi104.stdout: Creating ceph-iscsi config... 2024-09-21T12:35:42.183 INFO:journalctl@ceph.mon.b.smithi104.stdout: Write file: /var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/iscsi.foo.smithi079.vlhrek/iscsi-gateway.cfg 2024-09-21T12:35:42.183 INFO:journalctl@ceph.mon.b.smithi104.stdout: Write file: /var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/iscsi.foo.smithi079.vlhrek/tcmu-runner-entrypoint.sh 2024-09-21T12:35:42.183 INFO:journalctl@ceph.mon.b.smithi104.stdout: Non-zero exit code 1 from systemctl restart ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek 2024-09-21T12:35:42.183 INFO:journalctl@ceph.mon.b.smithi104.stdout: systemctl: stderr Job for ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service failed because the control process exited with error code. 2024-09-21T12:35:42.183 INFO:journalctl@ceph.mon.b.smithi104.stdout: systemctl: stderr See "systemctl status ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service" and "journalctl -xeu ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service" for details. 2024-09-21T12:35:42.183 INFO:journalctl@ceph.mon.b.smithi104.stdout: Traceback (most recent call last): 2024-09-21T12:35:42.183 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-21T12:35:42.183 INFO:journalctl@ceph.mon.b.smithi104.stdout: return _run_code(code, main_globals, None, 2024-09-21T12:35:42.184 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-21T12:35:42.184 INFO:journalctl@ceph.mon.b.smithi104.stdout: exec(code, run_globals) 2024-09-21T12:35:42.184 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 5628, in 2024-09-21T12:35:42.184 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 5616, in main 2024-09-21T12:35:42.184 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 3090, in command_deploy_from 2024-09-21T12:35:42.184 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 3125, in _common_deploy 2024-09-21T12:35:42.184 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 3145, in _deploy_daemon_container 2024-09-21T12:35:42.184 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 1147, in deploy_daemon 2024-09-21T12:35:42.184 INFO:journalctl@ceph.mon.b.smithi104.stdout: File "/tmp/tmptpwpfqo2.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-21T12:35:42.184 INFO:journalctl@ceph.mon.b.smithi104.stdout: RuntimeError: Failed command: systemctl restart ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek: Job for ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service failed because the control process exited with error code. 2024-09-21T12:35:42.185 INFO:journalctl@ceph.mon.b.smithi104.stdout: See "systemctl status ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service" and "journalctl -xeu ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service" for details. 2024-09-21T12:35:42.185 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:41 smithi104 ceph-mon[25639]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:35:42.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:41 smithi079 ceph-mon[23699]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi079.vlhrek ... 2024-09-21T12:35:42.192 INFO:journalctl@ceph.mon.a.smithi079.stdout: Creating ceph-iscsi config... 2024-09-21T12:35:42.192 INFO:journalctl@ceph.mon.a.smithi079.stdout: Write file: /var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/iscsi.foo.smithi079.vlhrek/iscsi-gateway.cfg 2024-09-21T12:35:42.192 INFO:journalctl@ceph.mon.a.smithi079.stdout: Write file: /var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/iscsi.foo.smithi079.vlhrek/tcmu-runner-entrypoint.sh 2024-09-21T12:35:42.192 INFO:journalctl@ceph.mon.a.smithi079.stdout: Non-zero exit code 1 from systemctl restart ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek 2024-09-21T12:35:42.192 INFO:journalctl@ceph.mon.a.smithi079.stdout: systemctl: stderr Job for ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service failed because the control process exited with error code. 2024-09-21T12:35:42.192 INFO:journalctl@ceph.mon.a.smithi079.stdout: systemctl: stderr See "systemctl status ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service" and "journalctl -xeu ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service" for details. 2024-09-21T12:35:42.192 INFO:journalctl@ceph.mon.a.smithi079.stdout: Traceback (most recent call last): 2024-09-21T12:35:42.192 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-21T12:35:42.192 INFO:journalctl@ceph.mon.a.smithi079.stdout: return _run_code(code, main_globals, None, 2024-09-21T12:35:42.193 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-21T12:35:42.193 INFO:journalctl@ceph.mon.a.smithi079.stdout: exec(code, run_globals) 2024-09-21T12:35:42.193 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 5628, in 2024-09-21T12:35:42.193 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 5616, in main 2024-09-21T12:35:42.193 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 3090, in command_deploy_from 2024-09-21T12:35:42.193 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 3125, in _common_deploy 2024-09-21T12:35:42.193 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 3145, in _deploy_daemon_container 2024-09-21T12:35:42.193 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 1147, in deploy_daemon 2024-09-21T12:35:42.193 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/tmp/tmptpwpfqo2.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-21T12:35:42.193 INFO:journalctl@ceph.mon.a.smithi079.stdout: RuntimeError: Failed command: systemctl restart ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek: Job for ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service failed because the control process exited with error code. 2024-09-21T12:35:42.193 INFO:journalctl@ceph.mon.a.smithi079.stdout: See "systemctl status ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service" and "journalctl -xeu ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service" for details. 2024-09-21T12:35:42.193 INFO:journalctl@ceph.mon.a.smithi079.stdout: Traceback (most recent call last): 2024-09-21T12:35:42.193 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1164, in _check_daemons 2024-09-21T12:35:42.193 INFO:journalctl@ceph.mon.a.smithi079.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-09-21T12:35:42.194 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2610, in _daemon_action 2024-09-21T12:35:42.194 INFO:journalctl@ceph.mon.a.smithi079.stdout: return self.wait_async( 2024-09-21T12:35:42.194 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 878, in wait_async 2024-09-21T12:35:42.194 INFO:journalctl@ceph.mon.a.smithi079.stdout: return self.event_loop.get_result(coro, timeout) 2024-09-21T12:35:42.194 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-09-21T12:35:42.194 INFO:journalctl@ceph.mon.a.smithi079.stdout: return future.result(timeout) 2024-09-21T12:35:42.194 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/lib64/python3.9/concurrent/futures/_base.py", line 446, in result 2024-09-21T12:35:42.194 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:41 smithi079 ceph-mon[30023]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi079.vlhrek ... 2024-09-21T12:35:42.194 INFO:journalctl@ceph.mon.c.smithi079.stdout: Creating ceph-iscsi config... 2024-09-21T12:35:42.194 INFO:journalctl@ceph.mon.c.smithi079.stdout: Write file: /var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/iscsi.foo.smithi079.vlhrek/iscsi-gateway.cfg 2024-09-21T12:35:42.195 INFO:journalctl@ceph.mon.c.smithi079.stdout: Write file: /var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/iscsi.foo.smithi079.vlhrek/tcmu-runner-entrypoint.sh 2024-09-21T12:35:42.195 INFO:journalctl@ceph.mon.c.smithi079.stdout: Non-zero exit code 1 from systemctl restart ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek 2024-09-21T12:35:42.195 INFO:journalctl@ceph.mon.c.smithi079.stdout: systemctl: stderr Job for ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service failed because the control process exited with error code. 2024-09-21T12:35:42.195 INFO:journalctl@ceph.mon.c.smithi079.stdout: systemctl: stderr See "systemctl status ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service" and "journalctl -xeu ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service" for details. 2024-09-21T12:35:42.195 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2024-09-21T12:35:42.195 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-21T12:35:42.195 INFO:journalctl@ceph.mon.c.smithi079.stdout: return _run_code(code, main_globals, None, 2024-09-21T12:35:42.195 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-21T12:35:42.195 INFO:journalctl@ceph.mon.c.smithi079.stdout: exec(code, run_globals) 2024-09-21T12:35:42.195 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 5628, in 2024-09-21T12:35:42.195 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 5616, in main 2024-09-21T12:35:42.195 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 3090, in command_deploy_from 2024-09-21T12:35:42.196 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 3125, in _common_deploy 2024-09-21T12:35:42.196 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 3145, in _deploy_daemon_container 2024-09-21T12:35:42.196 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 1147, in deploy_daemon 2024-09-21T12:35:42.196 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/tmp/tmptpwpfqo2.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-21T12:35:42.196 INFO:journalctl@ceph.mon.c.smithi079.stdout: RuntimeError: Failed command: systemctl restart ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek: Job for ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service failed because the control process exited with error code. 2024-09-21T12:35:42.196 INFO:journalctl@ceph.mon.c.smithi079.stdout: See "systemctl status ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service" and "journalctl -xeu ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service" for details. 2024-09-21T12:35:42.196 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2024-09-21T12:35:42.196 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1164, in _check_daemons 2024-09-21T12:35:42.196 INFO:journalctl@ceph.mon.c.smithi079.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-09-21T12:35:42.196 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2610, in _daemon_action 2024-09-21T12:35:42.196 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.wait_async( 2024-09-21T12:35:42.197 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 878, in wait_async 2024-09-21T12:35:42.197 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.event_loop.get_result(coro, timeout) 2024-09-21T12:35:42.197 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-09-21T12:35:42.197 INFO:journalctl@ceph.mon.c.smithi079.stdout: return future.result(timeout) 2024-09-21T12:35:42.197 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.9/concurrent/futures/_base.py", line 446, in result 2024-09-21T12:35:42.197 INFO:journalctl@ceph.mon.c.smithi079.stdout: return self.__get_result() 2024-09-21T12:35:42.197 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/lib64/python3.9/concurrent/futures/_base.py", line 391, in __get_result 2024-09-21T12:35:42.197 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise self._exception 2024-09-21T12:35:42.198 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1400, in _create_daemon 2024-09-21T12:35:42.198 INFO:journalctl@ceph.mon.c.smithi079.stdout: out, err, code = await self._run_cephadm( 2024-09-21T12:35:42.198 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1743, in _run_cephadm 2024-09-21T12:35:42.198 INFO:journalctl@ceph.mon.c.smithi079.stdout: raise OrchestratorError( 2024-09-21T12:35:42.198 INFO:journalctl@ceph.mon.c.smithi079.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi079.vlhrek ... 2024-09-21T12:35:42.198 INFO:journalctl@ceph.mon.c.smithi079.stdout: Creating ceph-iscsi config... 2024-09-21T12:35:42.198 INFO:journalctl@ceph.mon.c.smithi079.stdout: Write file: /var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/iscsi.foo.smithi079.vlhrek/iscsi-gateway.cfg 2024-09-21T12:35:42.198 INFO:journalctl@ceph.mon.c.smithi079.stdout: Write file: /var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/iscsi.foo.smithi079.vlhrek/tcmu-runner-entrypoint.sh 2024-09-21T12:35:42.199 INFO:journalctl@ceph.mon.c.smithi079.stdout: Non-zero exit code 1 from systemctl restart ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek 2024-09-21T12:35:42.199 INFO:journalctl@ceph.mon.c.smithi079.stdout: systemctl: stderr Job for ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service failed because the control process exited with error code. 2024-09-21T12:35:42.199 INFO:journalctl@ceph.mon.c.smithi079.stdout: systemctl: stderr See "systemctl status ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service" and "journalctl -xeu ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service" for details. 2024-09-21T12:35:42.199 INFO:journalctl@ceph.mon.c.smithi079.stdout: Traceback (most recent call last): 2024-09-21T12:35:42.199 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-21T12:35:42.199 INFO:journalctl@ceph.mon.c.smithi079.stdout: return _run_code(code, main_globals, None, 2024-09-21T12:35:42.199 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-21T12:35:42.199 INFO:journalctl@ceph.mon.c.smithi079.stdout: exec(code, run_globals) 2024-09-21T12:35:42.199 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 5628, in 2024-09-21T12:35:42.199 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 5616, in main 2024-09-21T12:35:42.199 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 3090, in command_deploy_from 2024-09-21T12:35:42.200 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 3125, in _common_deploy 2024-09-21T12:35:42.200 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 3145, in _deploy_daemon_container 2024-09-21T12:35:42.200 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 1147, in deploy_daemon 2024-09-21T12:35:42.200 INFO:journalctl@ceph.mon.c.smithi079.stdout: File "/tmp/tmptpwpfqo2.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-21T12:35:42.200 INFO:journalctl@ceph.mon.c.smithi079.stdout: RuntimeError: Failed command: systemctl restart ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek: Job for ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service failed because the control process exited with error code. 2024-09-21T12:35:42.200 INFO:journalctl@ceph.mon.c.smithi079.stdout: See "systemctl status ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service" and "journalctl -xeu ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service" for details. 2024-09-21T12:35:42.200 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:41 smithi079 ceph-mon[30023]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:35:42.200 INFO:journalctl@ceph.mon.a.smithi079.stdout: return self.__get_result() 2024-09-21T12:35:42.201 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/lib64/python3.9/concurrent/futures/_base.py", line 391, in __get_result 2024-09-21T12:35:42.201 INFO:journalctl@ceph.mon.a.smithi079.stdout: raise self._exception 2024-09-21T12:35:42.201 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1400, in _create_daemon 2024-09-21T12:35:42.201 INFO:journalctl@ceph.mon.a.smithi079.stdout: out, err, code = await self._run_cephadm( 2024-09-21T12:35:42.201 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1743, in _run_cephadm 2024-09-21T12:35:42.201 INFO:journalctl@ceph.mon.a.smithi079.stdout: raise OrchestratorError( 2024-09-21T12:35:42.201 INFO:journalctl@ceph.mon.a.smithi079.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi079.vlhrek ... 2024-09-21T12:35:42.201 INFO:journalctl@ceph.mon.a.smithi079.stdout: Creating ceph-iscsi config... 2024-09-21T12:35:42.201 INFO:journalctl@ceph.mon.a.smithi079.stdout: Write file: /var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/iscsi.foo.smithi079.vlhrek/iscsi-gateway.cfg 2024-09-21T12:35:42.201 INFO:journalctl@ceph.mon.a.smithi079.stdout: Write file: /var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/iscsi.foo.smithi079.vlhrek/tcmu-runner-entrypoint.sh 2024-09-21T12:35:42.202 INFO:journalctl@ceph.mon.a.smithi079.stdout: Non-zero exit code 1 from systemctl restart ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek 2024-09-21T12:35:42.202 INFO:journalctl@ceph.mon.a.smithi079.stdout: systemctl: stderr Job for ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service failed because the control process exited with error code. 2024-09-21T12:35:42.202 INFO:journalctl@ceph.mon.a.smithi079.stdout: systemctl: stderr See "systemctl status ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service" and "journalctl -xeu ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service" for details. 2024-09-21T12:35:42.202 INFO:journalctl@ceph.mon.a.smithi079.stdout: Traceback (most recent call last): 2024-09-21T12:35:42.202 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-21T12:35:42.202 INFO:journalctl@ceph.mon.a.smithi079.stdout: return _run_code(code, main_globals, None, 2024-09-21T12:35:42.202 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-21T12:35:42.202 INFO:journalctl@ceph.mon.a.smithi079.stdout: exec(code, run_globals) 2024-09-21T12:35:42.202 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 5628, in 2024-09-21T12:35:42.203 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 5616, in main 2024-09-21T12:35:42.203 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 3090, in command_deploy_from 2024-09-21T12:35:42.203 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 3125, in _common_deploy 2024-09-21T12:35:42.203 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 3145, in _deploy_daemon_container 2024-09-21T12:35:42.203 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 1147, in deploy_daemon 2024-09-21T12:35:42.203 INFO:journalctl@ceph.mon.a.smithi079.stdout: File "/tmp/tmptpwpfqo2.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-21T12:35:42.203 INFO:journalctl@ceph.mon.a.smithi079.stdout: RuntimeError: Failed command: systemctl restart ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek: Job for ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service failed because the control process exited with error code. 2024-09-21T12:35:42.203 INFO:journalctl@ceph.mon.a.smithi079.stdout: See "systemctl status ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service" and "journalctl -xeu ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service" for details. 2024-09-21T12:35:42.203 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:41 smithi079 ceph-mon[23699]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:35:43.064 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:42 smithi079 ceph-mon[23699]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 965 B/s rd, 1 op/s 2024-09-21T12:35:43.064 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:42 smithi079 ceph-mon[30023]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 965 B/s rd, 1 op/s 2024-09-21T12:35:43.176 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:42 smithi104 ceph-mon[25639]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 965 B/s rd, 1 op/s 2024-09-21T12:35:43.977 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:43 smithi079 ceph-mon[23699]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-21T12:35:43.977 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:43 smithi079 ceph-mon[23699]: from='mgr.25054 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-21T12:35:43.977 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:43 smithi079 ceph-mon[23699]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-21T12:35:43.977 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:43 smithi079 ceph-mon[23699]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:35:43.978 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:43 smithi079 ceph-mon[30023]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-21T12:35:43.978 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:43 smithi079 ceph-mon[30023]: from='mgr.25054 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-21T12:35:43.978 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:43 smithi079 ceph-mon[30023]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-21T12:35:43.978 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:43 smithi079 ceph-mon[30023]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:35:44.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:43 smithi104 ceph-mon[25639]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-21T12:35:44.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:43 smithi104 ceph-mon[25639]: from='mgr.25054 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-21T12:35:44.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:43 smithi104 ceph-mon[25639]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-21T12:35:44.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:43 smithi104 ceph-mon[25639]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:35:45.176 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:44 smithi104 ceph-mon[25639]: Upgrade: Updating mgr.y 2024-09-21T12:35:45.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:44 smithi104 ceph-mon[25639]: Deploying daemon mgr.y on smithi079 2024-09-21T12:35:45.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:44 smithi104 ceph-mon[25639]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-09-21T12:35:45.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:44 smithi104 ceph-mon[25639]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T12:35:45.181 INFO:teuthology.orchestra.run.smithi079.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2024-09-21T12:35:45.181 INFO:teuthology.orchestra.run.smithi079.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-09-21T12:35:45.181 INFO:teuthology.orchestra.run.smithi079.stdout: daemon iscsi.foo.smithi079.vlhrek on smithi079 is in unknown state 2024-09-21T12:35:45.181 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:44 smithi079 ceph-mon[23699]: Upgrade: Updating mgr.y 2024-09-21T12:35:45.182 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:44 smithi079 ceph-mon[23699]: Deploying daemon mgr.y on smithi079 2024-09-21T12:35:45.182 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:44 smithi079 ceph-mon[23699]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-09-21T12:35:45.182 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:44 smithi079 ceph-mon[23699]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T12:35:45.182 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:44 smithi079 ceph-mon[30023]: Upgrade: Updating mgr.y 2024-09-21T12:35:45.182 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:44 smithi079 ceph-mon[30023]: Deploying daemon mgr.y on smithi079 2024-09-21T12:35:45.182 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:44 smithi079 ceph-mon[30023]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-09-21T12:35:45.182 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:44 smithi079 ceph-mon[30023]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T12:35:45.942 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:45 smithi079 ceph-mon[23699]: from='client.? 172.21.15.79:0/3483964625' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-21T12:35:45.942 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:45 smithi079 ceph-mon[30023]: from='client.? 172.21.15.79:0/3483964625' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-21T12:35:46.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:45 smithi104 ceph-mon[25639]: from='client.? 172.21.15.79:0/3483964625' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-21T12:35:46.227 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 33168022-7814-11ef-baf6-efdc52797490 -e sha1=441784946648ff53dd310293f9dd752afeeea72a -- bash -c 'ceph versions | jq -e '"'"'.overall | length == 1'"'"'' 2024-09-21T12:35:46.252 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:35:46 smithi079 systemd[1]: Stopping Ceph mgr.y for 33168022-7814-11ef-baf6-efdc52797490... 2024-09-21T12:35:46.692 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:35:46 smithi079 podman[78503]: 2024-09-21 12:35:46.479700721 +0000 UTC m=+0.136347511 container died b129d767810ff43a3a72917eb494d55dcf5aa1e366fc560c493701eb5423f133 (image=quay.ceph.io/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y, RELEASE=wip-guits-main-2024-09-21-0842-4417849, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-09-21T12:35:47.108 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:46 smithi079 ceph-mon[23699]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T12:35:47.108 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:46 smithi079 ceph-mon[23699]: from='mgr.25054 ' entity='mgr.x' 2024-09-21T12:35:47.108 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:46 smithi079 ceph-mon[23699]: from='mgr.25054 ' entity='mgr.x' 2024-09-21T12:35:47.108 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:46 smithi079 ceph-mon[23699]: from='mgr.25054 ' entity='mgr.x' 2024-09-21T12:35:47.108 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:46 smithi079 ceph-mon[30023]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T12:35:47.108 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:46 smithi079 ceph-mon[30023]: from='mgr.25054 ' entity='mgr.x' 2024-09-21T12:35:47.108 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:46 smithi079 ceph-mon[30023]: from='mgr.25054 ' entity='mgr.x' 2024-09-21T12:35:47.108 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:46 smithi079 ceph-mon[30023]: from='mgr.25054 ' entity='mgr.x' 2024-09-21T12:35:47.109 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:35:46 smithi079 podman[78503]: 2024-09-21 12:35:46.91395278 +0000 UTC m=+0.570599557 container cleanup b129d767810ff43a3a72917eb494d55dcf5aa1e366fc560c493701eb5423f133 (image=quay.ceph.io/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-guits-main-2024-09-21-0842-4417849, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T12:35:47.109 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:35:46 smithi079 bash[78503]: ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y 2024-09-21T12:35:47.176 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:46 smithi104 ceph-mon[25639]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-21T12:35:47.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:46 smithi104 ceph-mon[25639]: from='mgr.25054 ' entity='mgr.x' 2024-09-21T12:35:47.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:46 smithi104 ceph-mon[25639]: from='mgr.25054 ' entity='mgr.x' 2024-09-21T12:35:47.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:46 smithi104 ceph-mon[25639]: from='mgr.25054 ' entity='mgr.x' 2024-09-21T12:35:47.419 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:35:47 smithi079 podman[78560]: 2024-09-21 12:35:47.107081297 +0000 UTC m=+0.626532708 container remove b129d767810ff43a3a72917eb494d55dcf5aa1e366fc560c493701eb5423f133 (image=quay.ceph.io/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-main-2024-09-21-0842-4417849, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True) 2024-09-21T12:35:47.419 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:35:47 smithi079 systemd[1]: ceph-33168022-7814-11ef-baf6-efdc52797490@mgr.y.service: Main process exited, code=exited, status=143/n/a 2024-09-21T12:35:47.779 INFO:teuthology.orchestra.run.smithi079.stderr:Inferring config /var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T12:35:48.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:47 smithi104 ceph-mon[25639]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 385 B/s rd, 0 op/s 2024-09-21T12:35:48.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:47 smithi104 ceph-mon[25639]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 474 B/s rd, 0 op/s 2024-09-21T12:35:48.191 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:47 smithi079 ceph-mon[23699]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 385 B/s rd, 0 op/s 2024-09-21T12:35:48.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:47 smithi079 ceph-mon[23699]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 474 B/s rd, 0 op/s 2024-09-21T12:35:48.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:47 smithi079 ceph-mon[30023]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 385 B/s rd, 0 op/s 2024-09-21T12:35:48.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:47 smithi079 ceph-mon[30023]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 255 MiB used, 715 GiB / 715 GiB avail; 474 B/s rd, 0 op/s 2024-09-21T12:35:48.804 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:35:48 smithi079 podman[78645]: 2024-09-21 12:35:48.509615957 +0000 UTC m=+0.725881801 container died e76a72b09d14c73fa0f559403f9fa3e7dc6317b82444f88888d9e9fa2f19e638 (image=quay.ceph.io/ceph-ci/ceph:reef, name=hungry_colden, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, RELEASE=reef-6a1bdf8, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-09-21T12:35:48.804 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:35:48 smithi079 systemd[1]: ceph-33168022-7814-11ef-baf6-efdc52797490@mgr.y.service: Failed with result 'exit-code'. 2024-09-21T12:35:48.804 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:35:48 smithi079 systemd[1]: Stopped Ceph mgr.y for 33168022-7814-11ef-baf6-efdc52797490. 2024-09-21T12:35:48.804 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:35:48 smithi079 systemd[1]: ceph-33168022-7814-11ef-baf6-efdc52797490@mgr.y.service: Consumed 6.276s CPU time. 2024-09-21T12:35:49.191 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:35:48 smithi079 systemd[1]: Starting Ceph mgr.y for 33168022-7814-11ef-baf6-efdc52797490... 2024-09-21T12:35:49.820 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:35:49 smithi079 podman[78805]: 2024-09-21 12:35:49.624488457 +0000 UTC m=+0.104446136 container create b64e6b3d276a83a384a57be7e875c3f6ec7e0825d4ff1d1638c1510d3617fbc7 (image=quay.ceph.io/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.37.2, RELEASE=wip-guits-main-2024-09-21-0842-4417849, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD) 2024-09-21T12:35:49.821 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:35:49 smithi079 podman[78805]: 2024-09-21 12:35:49.538248717 +0000 UTC m=+0.018206409 image pull b415816db8b41f544cdfdc71d193ff8b1ee008bf7b0ec4f64da8eb8f71b62e85 quay.ceph.io/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5 2024-09-21T12:35:50.104 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:49 smithi079 ceph-mon[23699]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 255 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:35:50.104 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:49 smithi079 ceph-mon[30023]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 255 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:35:50.105 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:35:50 smithi079 podman[78805]: 2024-09-21 12:35:50.020337647 +0000 UTC m=+0.500295327 container init b64e6b3d276a83a384a57be7e875c3f6ec7e0825d4ff1d1638c1510d3617fbc7 (image=quay.ceph.io/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y, RELEASE=wip-guits-main-2024-09-21-0842-4417849, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS) 2024-09-21T12:35:50.105 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:35:50 smithi079 podman[78805]: 2024-09-21 12:35:50.023382134 +0000 UTC m=+0.503339823 container start b64e6b3d276a83a384a57be7e875c3f6ec7e0825d4ff1d1638c1510d3617fbc7 (image=quay.ceph.io/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y, RELEASE=wip-guits-main-2024-09-21-0842-4417849, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, ceph=True, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T12:35:50.105 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:35:50 smithi079 ceph-mgr[78847]: -- 172.21.15.79:0/1020107435 <== mon.2 v2:172.21.15.79:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55a046b6b0e0 con 0x55a046b48800 2024-09-21T12:35:50.176 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:49 smithi104 ceph-mon[25639]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 255 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:35:50.178 INFO:teuthology.orchestra.run.smithi079.stdout:false 2024-09-21T12:35:50.441 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:35:50 smithi079 bash[78805]: b64e6b3d276a83a384a57be7e875c3f6ec7e0825d4ff1d1638c1510d3617fbc7 2024-09-21T12:35:50.442 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:35:50 smithi079 systemd[1]: Started Ceph mgr.y for 33168022-7814-11ef-baf6-efdc52797490. 2024-09-21T12:35:51.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:50 smithi104 ceph-mon[25639]: from='client.? 172.21.15.79:0/2622757408' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T12:35:51.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:50 smithi104 ceph-mon[25639]: from='mgr.25054 ' entity='mgr.x' 2024-09-21T12:35:51.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:50 smithi104 ceph-mon[25639]: from='mgr.25054 ' entity='mgr.x' 2024-09-21T12:35:51.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:50 smithi104 ceph-mon[25639]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:35:51.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:50 smithi104 ceph-mon[25639]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:35:51.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:50 smithi104 ceph-mon[25639]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:35:51.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:50 smithi104 ceph-mon[25639]: from='mgr.25054 ' entity='mgr.x' 2024-09-21T12:35:51.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:50 smithi104 ceph-mon[25639]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi079.vlhrek", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-21T12:35:51.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:50 smithi104 ceph-mon[25639]: from='mgr.25054 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi079.vlhrek", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-21T12:35:51.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:50 smithi104 ceph-mon[25639]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:35:51.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:50 smithi079 ceph-mon[23699]: from='client.? 172.21.15.79:0/2622757408' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T12:35:51.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:50 smithi079 ceph-mon[23699]: from='mgr.25054 ' entity='mgr.x' 2024-09-21T12:35:51.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:50 smithi079 ceph-mon[23699]: from='mgr.25054 ' entity='mgr.x' 2024-09-21T12:35:51.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:50 smithi079 ceph-mon[23699]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:35:51.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:50 smithi079 ceph-mon[23699]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:35:51.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:50 smithi079 ceph-mon[23699]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:35:51.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:50 smithi079 ceph-mon[23699]: from='mgr.25054 ' entity='mgr.x' 2024-09-21T12:35:51.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:50 smithi079 ceph-mon[23699]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi079.vlhrek", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-21T12:35:51.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:50 smithi079 ceph-mon[23699]: from='mgr.25054 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi079.vlhrek", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-21T12:35:51.192 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:50 smithi079 ceph-mon[23699]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:35:51.192 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:50 smithi079 ceph-mon[30023]: from='client.? 172.21.15.79:0/2622757408' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-21T12:35:51.193 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:50 smithi079 ceph-mon[30023]: from='mgr.25054 ' entity='mgr.x' 2024-09-21T12:35:51.193 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:50 smithi079 ceph-mon[30023]: from='mgr.25054 ' entity='mgr.x' 2024-09-21T12:35:51.193 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:50 smithi079 ceph-mon[30023]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T12:35:51.193 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:50 smithi079 ceph-mon[30023]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:35:51.193 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:50 smithi079 ceph-mon[30023]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T12:35:51.193 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:50 smithi079 ceph-mon[30023]: from='mgr.25054 ' entity='mgr.x' 2024-09-21T12:35:51.193 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:50 smithi079 ceph-mon[30023]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi079.vlhrek", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-21T12:35:51.193 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:50 smithi079 ceph-mon[30023]: from='mgr.25054 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi079.vlhrek", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-21T12:35:51.193 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:50 smithi079 ceph-mon[30023]: from='mgr.25054 172.21.15.104:0/258127029' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T12:35:51.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:35:51.254 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_441784946648ff53dd310293f9dd752afeeea72a/qa/tasks/cephadm.py", line 1525, in shell _shell( File "/home/teuthworker/src/git.ceph.com_ceph-c_441784946648ff53dd310293f9dd752afeeea72a/qa/tasks/cephadm.py", line 111, in _shell return remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi079 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 33168022-7814-11ef-baf6-efdc52797490 -e sha1=441784946648ff53dd310293f9dd752afeeea72a -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | length == 1\'"\'"\'\'' 2024-09-21T12:35:51.417 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=96bb007e133b49abbf635ac5cb77d7ff Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_441784946648ff53dd310293f9dd752afeeea72a/qa/tasks/cephadm.py", line 1525, in shell _shell( File "/home/teuthworker/src/git.ceph.com_ceph-c_441784946648ff53dd310293f9dd752afeeea72a/qa/tasks/cephadm.py", line 111, in _shell return remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi079 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 33168022-7814-11ef-baf6-efdc52797490 -e sha1=441784946648ff53dd310293f9dd752afeeea72a -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | length == 1\'"\'"\'\'' 2024-09-21T12:35:51.419 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-09-21T12:35:51.429 INFO:tasks.cephadm:Teardown begin 2024-09-21T12:35:51.429 DEBUG:teuthology.orchestra.run.smithi079:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-21T12:35:51.463 DEBUG:teuthology.orchestra.run.smithi104:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-21T12:35:51.496 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-09-21T12:35:51.497 DEBUG:teuthology.orchestra.run.smithi079:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-09-21T12:35:51.518 DEBUG:teuthology.orchestra.run.smithi104:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-09-21T12:35:51.553 INFO:tasks.cephadm:Stopping all daemons... 2024-09-21T12:35:51.554 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-09-21T12:35:51.554 DEBUG:teuthology.orchestra.run.smithi079:> sudo systemctl stop ceph-33168022-7814-11ef-baf6-efdc52797490@mon.a 2024-09-21T12:35:51.837 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:51 smithi079 systemd[1]: Stopping Ceph mon.a for 33168022-7814-11ef-baf6-efdc52797490... 2024-09-21T12:35:52.131 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:51 smithi079 ceph-mon[23699]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 255 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:35:52.131 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:51 smithi079 ceph-mon[23699]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 255 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:35:52.131 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:51 smithi079 ceph-mon[23699]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-21T12:35:52.131 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:51 smithi079 ceph-mon[23699]: Reconfiguring iscsi.foo.smithi079.vlhrek (dependencies changed)... 2024-09-21T12:35:52.131 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:51 smithi079 ceph-mon[23699]: Reconfiguring daemon iscsi.foo.smithi079.vlhrek on smithi079 2024-09-21T12:35:52.132 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:51 smithi079 ceph-mon[30023]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 255 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:35:52.132 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:51 smithi079 ceph-mon[30023]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 255 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:35:52.132 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:51 smithi079 ceph-mon[30023]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-21T12:35:52.132 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:51 smithi079 ceph-mon[30023]: Reconfiguring iscsi.foo.smithi079.vlhrek (dependencies changed)... 2024-09-21T12:35:52.132 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:51 smithi079 ceph-mon[30023]: Reconfiguring daemon iscsi.foo.smithi079.vlhrek on smithi079 2024-09-21T12:35:52.177 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:51 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: ::ffff:172.21.15.104 - - [21/Sep/2024:12:35:51] "GET /metrics HTTP/1.1" 200 38582 "" "Prometheus/2.43.0" 2024-09-21T12:35:52.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:51 smithi104 ceph-mon[25639]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 255 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:35:52.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:51 smithi104 ceph-mon[25639]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 255 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:35:52.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:51 smithi104 ceph-mon[25639]: Metadata not up to date on all hosts. Skipping non agent specs 2024-09-21T12:35:52.177 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:51 smithi104 ceph-mon[25639]: Reconfiguring iscsi.foo.smithi079.vlhrek (dependencies changed)... 2024-09-21T12:35:52.178 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:51 smithi104 ceph-mon[25639]: Reconfiguring daemon iscsi.foo.smithi079.vlhrek on smithi079 2024-09-21T12:35:54.188 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:53 smithi079 ceph-mon[23699]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 255 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:35:54.188 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:53 smithi079 ceph-mon[23699]: from='client.? 172.21.15.79:0/3259182178' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-21T12:35:54.188 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:53 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mon-a[23695]: 2024-09-21T12:35:53.977+0000 7fbf9b954640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T12:35:54.188 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:53 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mon-a[23695]: 2024-09-21T12:35:53.977+0000 7fbf9b954640 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-09-21T12:35:54.189 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:53 smithi079 podman[79301]: 2024-09-21 12:35:53.998632998 +0000 UTC m=+0.820348359 container died 794d66ac29c5f080f3d25c707ee3cb754cd838c130700defb586dd0788bdd96a (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-33168022-7814-11ef-baf6-efdc52797490-mon-a, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=reef-6a1bdf8) 2024-09-21T12:35:54.189 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:53 smithi079 ceph-mon[30023]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 255 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:35:54.189 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:53 smithi079 ceph-mon[30023]: from='client.? 172.21.15.79:0/3259182178' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-21T12:35:54.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:53 smithi104 ceph-mon[25639]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 255 MiB used, 715 GiB / 715 GiB avail 2024-09-21T12:35:54.427 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:53 smithi104 ceph-mon[25639]: from='client.? 172.21.15.79:0/3259182178' entity='client.iscsi.foo.smithi079.vlhrek' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-21T12:35:54.441 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:54 smithi079 podman[79301]: 2024-09-21 12:35:54.428160495 +0000 UTC m=+1.249875847 container cleanup 794d66ac29c5f080f3d25c707ee3cb754cd838c130700defb586dd0788bdd96a (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-33168022-7814-11ef-baf6-efdc52797490-mon-a, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, RELEASE=reef-6a1bdf8, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-09-21T12:35:54.441 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:54 smithi079 bash[79301]: ceph-33168022-7814-11ef-baf6-efdc52797490-mon-a 2024-09-21T12:35:55.320 INFO:journalctl@ceph.mon.a.smithi079.stdout:Sep 21 12:35:55 smithi079 podman[79398]: 2024-09-21 12:35:55.057631249 +0000 UTC m=+1.058439120 container remove 794d66ac29c5f080f3d25c707ee3cb754cd838c130700defb586dd0788bdd96a (image=quay.ceph.io/ceph-ci/ceph:reef, name=ceph-33168022-7814-11ef-baf6-efdc52797490-mon-a, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, ceph=True, CEPH_POINT_RELEASE=, RELEASE=reef-6a1bdf8) 2024-09-21T12:35:55.351 DEBUG:teuthology.orchestra.run.smithi079:> sudo pkill -f 'journalctl -f -n 0 -u ceph-33168022-7814-11ef-baf6-efdc52797490@mon.a.service' 2024-09-21T12:35:55.399 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T12:35:55.399 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-09-21T12:35:55.399 INFO:tasks.cephadm.mon.b:Stopping mon.c... 2024-09-21T12:35:55.400 DEBUG:teuthology.orchestra.run.smithi079:> sudo systemctl stop ceph-33168022-7814-11ef-baf6-efdc52797490@mon.c 2024-09-21T12:35:55.692 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:55 smithi079 systemd[1]: Stopping Ceph mon.c for 33168022-7814-11ef-baf6-efdc52797490... 2024-09-21T12:35:55.692 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:35:55 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[78820]: [21/Sep/2024:12:35:55] ENGINE Bus STARTING 2024-09-21T12:35:55.692 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:35:55 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[78820]: CherryPy Checker: 2024-09-21T12:35:55.692 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:35:55 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[78820]: The Application mounted at '' has an empty config. 2024-09-21T12:35:55.693 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:35:55 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[78820]: 2024-09-21T12:35:55.693 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:35:55 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[78820]: [21/Sep/2024:12:35:55] ENGINE Serving on http://:::9283 2024-09-21T12:35:55.693 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:35:55 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y[78820]: [21/Sep/2024:12:35:55] ENGINE Bus STARTED 2024-09-21T12:35:57.647 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:57 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mon-c[30019]: 2024-09-21T12:35:57.203+0000 7fe24ce53640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T12:35:57.647 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:57 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-mon-c[30019]: 2024-09-21T12:35:57.203+0000 7fe24ce53640 -1 mon.c@2(peon) e3 *** Got Signal Terminated *** 2024-09-21T12:35:57.647 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:57 smithi079 podman[79698]: 2024-09-21 12:35:57.36696042 +0000 UTC m=+1.021097173 container died 4b837a3d8ec3f5a95e11dac3edadd2bfc94e662bcb04ab8dc8360ea3677dc85c (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-33168022-7814-11ef-baf6-efdc52797490-mon-c, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=reef-6a1bdf8, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, org.label-schema.build-date=20240731, GIT_CLEAN=True) 2024-09-21T12:35:57.647 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:57 smithi079 podman[79698]: 2024-09-21 12:35:57.645838023 +0000 UTC m=+1.299974774 container cleanup 4b837a3d8ec3f5a95e11dac3edadd2bfc94e662bcb04ab8dc8360ea3677dc85c (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-33168022-7814-11ef-baf6-efdc52797490-mon-c, RELEASE=reef-6a1bdf8, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T12:35:57.898 INFO:journalctl@ceph.mon.c.smithi079.stdout:Sep 21 12:35:57 smithi079 bash[79698]: ceph-33168022-7814-11ef-baf6-efdc52797490-mon-c 2024-09-21T12:35:58.042 DEBUG:teuthology.orchestra.run.smithi079:> sudo pkill -f 'journalctl -f -n 0 -u ceph-33168022-7814-11ef-baf6-efdc52797490@mon.c.service' 2024-09-21T12:35:58.079 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T12:35:58.079 INFO:tasks.cephadm.mon.b:Stopped mon.c 2024-09-21T12:35:58.080 INFO:tasks.cephadm.mon.b:Stopping mon.b... 2024-09-21T12:35:58.080 DEBUG:teuthology.orchestra.run.smithi104:> sudo systemctl stop ceph-33168022-7814-11ef-baf6-efdc52797490@mon.b 2024-09-21T12:35:58.106 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:57 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: 2024-09-21T12:35:57.764+0000 7ff086688640 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-33168022-7814-11ef-baf6-efdc52797490-iscsi-foo-smithi079-vlhrek 2024-09-21T12:35:58.107 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:57 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: /usr/bin/podman: stderr Error: no such container ceph-33168022-7814-11ef-baf6-efdc52797490-iscsi-foo-smithi079-vlhrek 2024-09-21T12:35:58.107 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:57 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-33168022-7814-11ef-baf6-efdc52797490-iscsi.foo.smithi079.vlhrek 2024-09-21T12:35:58.107 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:57 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: /usr/bin/podman: stderr Error: no such container ceph-33168022-7814-11ef-baf6-efdc52797490-iscsi.foo.smithi079.vlhrek 2024-09-21T12:35:58.107 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:57 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: Reconfig daemon iscsi.foo.smithi079.vlhrek ... 2024-09-21T12:35:58.107 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:57 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: Creating ceph-iscsi config... 2024-09-21T12:35:58.107 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:57 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: Write file: /var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/iscsi.foo.smithi079.vlhrek/iscsi-gateway.cfg 2024-09-21T12:35:58.107 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:57 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: Write file: /var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/iscsi.foo.smithi079.vlhrek/tcmu-runner-entrypoint.sh 2024-09-21T12:35:58.107 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:57 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: Non-zero exit code 1 from systemctl restart ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek 2024-09-21T12:35:58.107 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:57 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: systemctl: stderr Job for ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service failed because the control process exited with error code. 2024-09-21T12:35:58.108 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:57 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: systemctl: stderr See "systemctl status ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service" and "journalctl -xeu ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service" for details. 2024-09-21T12:35:58.108 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:57 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: Traceback (most recent call last): 2024-09-21T12:35:58.108 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:57 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-21T12:35:58.108 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:57 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: return _run_code(code, main_globals, None, 2024-09-21T12:35:58.108 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:57 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-21T12:35:58.108 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:57 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: exec(code, run_globals) 2024-09-21T12:35:58.108 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:57 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 5628, in 2024-09-21T12:35:58.108 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:57 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 5616, in main 2024-09-21T12:35:58.108 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:57 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 3090, in command_deploy_from 2024-09-21T12:35:58.108 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:57 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 3125, in _common_deploy 2024-09-21T12:35:58.109 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:57 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 3145, in _deploy_daemon_container 2024-09-21T12:35:58.109 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:57 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 1147, in deploy_daemon 2024-09-21T12:35:58.109 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:57 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: File "/tmp/tmptpwpfqo2.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-21T12:35:58.109 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:57 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: RuntimeError: Failed command: systemctl restart ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek: Job for ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service failed because the control process exited with error code. 2024-09-21T12:35:58.109 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:57 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: See "systemctl status ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service" and "journalctl -xeu ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service" for details. 2024-09-21T12:35:58.109 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:57 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: Traceback (most recent call last): 2024-09-21T12:35:58.110 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:57 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1164, in _check_daemons 2024-09-21T12:35:58.110 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:57 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: self.mgr._daemon_action(daemon_spec, action=action) 2024-09-21T12:35:58.110 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:57 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2610, in _daemon_action 2024-09-21T12:35:58.110 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:57 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: return self.wait_async( 2024-09-21T12:35:58.110 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:57 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: File "/usr/share/ceph/mgr/cephadm/module.py", line 878, in wait_async 2024-09-21T12:35:58.110 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:57 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: return self.event_loop.get_result(coro, timeout) 2024-09-21T12:35:58.110 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:57 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-09-21T12:35:58.110 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:57 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: return future.result(timeout) 2024-09-21T12:35:58.111 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:57 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: File "/lib64/python3.9/concurrent/futures/_base.py", line 446, in result 2024-09-21T12:35:58.111 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:57 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: return self.__get_result() 2024-09-21T12:35:58.111 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:57 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: File "/lib64/python3.9/concurrent/futures/_base.py", line 391, in __get_result 2024-09-21T12:35:58.111 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:57 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: raise self._exception 2024-09-21T12:35:58.111 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:57 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1400, in _create_daemon 2024-09-21T12:35:58.111 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:57 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: out, err, code = await self._run_cephadm( 2024-09-21T12:35:58.111 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:57 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1743, in _run_cephadm 2024-09-21T12:35:58.111 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:57 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: raise OrchestratorError( 2024-09-21T12:35:58.111 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:57 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-33168022-7814-11ef-baf6-efdc52797490-iscsi-foo-smithi079-vlhrek 2024-09-21T12:35:58.112 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:57 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: /usr/bin/podman: stderr Error: no such container ceph-33168022-7814-11ef-baf6-efdc52797490-iscsi-foo-smithi079-vlhrek 2024-09-21T12:35:58.112 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:57 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-33168022-7814-11ef-baf6-efdc52797490-iscsi.foo.smithi079.vlhrek 2024-09-21T12:35:58.112 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:57 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: /usr/bin/podman: stderr Error: no such container ceph-33168022-7814-11ef-baf6-efdc52797490-iscsi.foo.smithi079.vlhrek 2024-09-21T12:35:58.112 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:57 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: Reconfig daemon iscsi.foo.smithi079.vlhrek ... 2024-09-21T12:35:58.112 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:57 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: Creating ceph-iscsi config... 2024-09-21T12:35:58.112 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:57 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: Write file: /var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/iscsi.foo.smithi079.vlhrek/iscsi-gateway.cfg 2024-09-21T12:35:58.112 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:57 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: Write file: /var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/iscsi.foo.smithi079.vlhrek/tcmu-runner-entrypoint.sh 2024-09-21T12:35:58.112 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:57 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: Non-zero exit code 1 from systemctl restart ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek 2024-09-21T12:35:58.112 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:57 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: systemctl: stderr Job for ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service failed because the control process exited with error code. 2024-09-21T12:35:58.113 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:57 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: systemctl: stderr See "systemctl status ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service" and "journalctl -xeu ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service" for details. 2024-09-21T12:35:58.113 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:57 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: Traceback (most recent call last): 2024-09-21T12:35:58.113 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:57 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-21T12:35:58.113 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:57 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: return _run_code(code, main_globals, None, 2024-09-21T12:35:58.113 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:57 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-21T12:35:58.113 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:57 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: exec(code, run_globals) 2024-09-21T12:35:58.113 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:57 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 5628, in 2024-09-21T12:35:58.113 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:57 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 5616, in main 2024-09-21T12:35:58.113 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:57 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 3090, in command_deploy_from 2024-09-21T12:35:58.114 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:57 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 3125, in _common_deploy 2024-09-21T12:35:58.114 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:57 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 3145, in _deploy_daemon_container 2024-09-21T12:35:58.114 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:57 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: File "/tmp/tmptpwpfqo2.cephadm.build/app/__main__.py", line 1147, in deploy_daemon 2024-09-21T12:35:58.114 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:57 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: File "/tmp/tmptpwpfqo2.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-21T12:35:58.114 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:57 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: RuntimeError: Failed command: systemctl restart ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek: Job for ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service failed because the control process exited with error code. 2024-09-21T12:35:58.114 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:35:57 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: See "systemctl status ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service" and "journalctl -xeu ceph-33168022-7814-11ef-baf6-efdc52797490@iscsi.foo.smithi079.vlhrek.service" for details. 2024-09-21T12:35:58.377 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:58 smithi104 systemd[1]: Stopping Ceph mon.b for 33168022-7814-11ef-baf6-efdc52797490... 2024-09-21T12:35:58.676 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:58 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mon-b[25635]: 2024-09-21T12:35:58.375+0000 7f195e737640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T12:35:58.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:58 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mon-b[25635]: 2024-09-21T12:35:58.375+0000 7f195e737640 -1 mon.b@1(peon) e3 *** Got Signal Terminated *** 2024-09-21T12:35:58.677 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:58 smithi104 podman[64662]: 2024-09-21 12:35:58.42672746 +0000 UTC m=+0.154621184 container died 460a6e8db654c089a736529f410a34461581257919fd6bf9bc711f1bdb5830fe (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-33168022-7814-11ef-baf6-efdc52797490-mon-b, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=reef-6a1bdf8, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731) 2024-09-21T12:35:59.078 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:58 smithi104 podman[64662]: 2024-09-21 12:35:58.850902322 +0000 UTC m=+0.578796047 container cleanup 460a6e8db654c089a736529f410a34461581257919fd6bf9bc711f1bdb5830fe (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-33168022-7814-11ef-baf6-efdc52797490-mon-b, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, ceph=True, RELEASE=reef-6a1bdf8, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T12:35:59.079 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:58 smithi104 bash[64662]: ceph-33168022-7814-11ef-baf6-efdc52797490-mon-b 2024-09-21T12:35:59.374 INFO:journalctl@ceph.mon.b.smithi104.stdout:Sep 21 12:35:59 smithi104 podman[64674]: 2024-09-21 12:35:59.077708971 +0000 UTC m=+0.641141478 container remove 460a6e8db654c089a736529f410a34461581257919fd6bf9bc711f1bdb5830fe (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-33168022-7814-11ef-baf6-efdc52797490-mon-b, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=reef-6a1bdf8, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-09-21T12:36:00.278 DEBUG:teuthology.orchestra.run.smithi104:> sudo pkill -f 'journalctl -f -n 0 -u ceph-33168022-7814-11ef-baf6-efdc52797490@mon.b.service' 2024-09-21T12:36:00.385 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T12:36:00.385 INFO:tasks.cephadm.mon.b:Stopped mon.b 2024-09-21T12:36:00.385 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-09-21T12:36:00.385 DEBUG:teuthology.orchestra.run.smithi079:> sudo systemctl stop ceph-33168022-7814-11ef-baf6-efdc52797490@mgr.y 2024-09-21T12:36:00.691 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:36:00 smithi079 systemd[1]: Stopping Ceph mgr.y for 33168022-7814-11ef-baf6-efdc52797490... 2024-09-21T12:36:01.191 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:36:00 smithi079 podman[80289]: 2024-09-21 12:36:00.918090692 +0000 UTC m=+0.165299250 container died b64e6b3d276a83a384a57be7e875c3f6ec7e0825d4ff1d1638c1510d3617fbc7 (image=quay.ceph.io/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-guits-main-2024-09-21-0842-4417849, io.buildah.version=1.37.2) 2024-09-21T12:36:01.589 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:36:01 smithi079 podman[80289]: 2024-09-21 12:36:01.295686397 +0000 UTC m=+0.542894946 container cleanup b64e6b3d276a83a384a57be7e875c3f6ec7e0825d4ff1d1638c1510d3617fbc7 (image=quay.ceph.io/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-guits-main-2024-09-21-0842-4417849, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.37.2) 2024-09-21T12:36:01.590 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:36:01 smithi079 bash[80289]: ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y 2024-09-21T12:36:01.927 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:36:01 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: ::ffff:172.21.15.104 - - [21/Sep/2024:12:36:01] "GET /metrics HTTP/1.1" 200 38582 "" "Prometheus/2.43.0" 2024-09-21T12:36:02.475 DEBUG:teuthology.orchestra.run.smithi079:> sudo pkill -f 'journalctl -f -n 0 -u ceph-33168022-7814-11ef-baf6-efdc52797490@mgr.y.service' 2024-09-21T12:36:02.498 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:36:02 smithi079 podman[80326]: 2024-09-21 12:36:02.242863068 +0000 UTC m=+1.316373519 container remove b64e6b3d276a83a384a57be7e875c3f6ec7e0825d4ff1d1638c1510d3617fbc7 (image=quay.ceph.io/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-y, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-guits-main-2024-09-21-0842-4417849, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-09-21T12:36:02.498 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:36:02 smithi079 systemd[1]: ceph-33168022-7814-11ef-baf6-efdc52797490@mgr.y.service: Main process exited, code=exited, status=143/n/a 2024-09-21T12:36:02.498 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:36:02 smithi079 systemd[1]: ceph-33168022-7814-11ef-baf6-efdc52797490@mgr.y.service: Failed with result 'exit-code'. 2024-09-21T12:36:02.499 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:36:02 smithi079 systemd[1]: Stopped Ceph mgr.y for 33168022-7814-11ef-baf6-efdc52797490. 2024-09-21T12:36:02.499 INFO:journalctl@ceph.mgr.y.smithi079.stdout:Sep 21 12:36:02 smithi079 systemd[1]: ceph-33168022-7814-11ef-baf6-efdc52797490@mgr.y.service: Consumed 6.308s CPU time. 2024-09-21T12:36:02.512 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T12:36:02.513 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-09-21T12:36:02.513 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-09-21T12:36:02.513 DEBUG:teuthology.orchestra.run.smithi104:> sudo systemctl stop ceph-33168022-7814-11ef-baf6-efdc52797490@mgr.x 2024-09-21T12:36:02.671 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:36:02 smithi104 systemd[1]: Stopping Ceph mgr.x for 33168022-7814-11ef-baf6-efdc52797490... 2024-09-21T12:36:02.927 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:36:02 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: 2024-09-21T12:36:02.870+0000 7ff0f33c3640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.x -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T12:36:02.927 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:36:02 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x[56348]: 2024-09-21T12:36:02.870+0000 7ff0f33c3640 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-09-21T12:36:02.927 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:36:02 smithi104 podman[65011]: 2024-09-21 12:36:02.911423072 +0000 UTC m=+0.173669905 container died 9038010f43b86a68e04abc5901b6b32af0e2e288e413ae8c9b8472a00239257b (image=quay.ceph.io/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-guits-main-2024-09-21-0842-4417849, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2) 2024-09-21T12:36:03.537 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:36:03 smithi104 podman[65011]: 2024-09-21 12:36:03.28104485 +0000 UTC m=+0.543291681 container cleanup 9038010f43b86a68e04abc5901b6b32af0e2e288e413ae8c9b8472a00239257b (image=quay.ceph.io/ceph-ci/ceph@sha256:f8fbbbcab363df772d74f326e361db3b4f83aa3233da1ebe9de08a83e748d9f5, name=ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-main-2024-09-21-0842-4417849, io.buildah.version=1.37.2, ceph=True) 2024-09-21T12:36:03.538 INFO:journalctl@ceph.mgr.x.smithi104.stdout:Sep 21 12:36:03 smithi104 bash[65011]: ceph-33168022-7814-11ef-baf6-efdc52797490-mgr-x 2024-09-21T12:36:03.670 DEBUG:teuthology.orchestra.run.smithi104:> sudo pkill -f 'journalctl -f -n 0 -u ceph-33168022-7814-11ef-baf6-efdc52797490@mgr.x.service' 2024-09-21T12:36:03.708 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T12:36:03.709 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-09-21T12:36:03.709 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-09-21T12:36:03.709 DEBUG:teuthology.orchestra.run.smithi079:> sudo systemctl stop ceph-33168022-7814-11ef-baf6-efdc52797490@osd.0 2024-09-21T12:36:04.022 INFO:journalctl@ceph.osd.0.smithi079.stdout:Sep 21 12:36:03 smithi079 systemd[1]: Stopping Ceph osd.0 for 33168022-7814-11ef-baf6-efdc52797490... 2024-09-21T12:36:04.441 INFO:journalctl@ceph.osd.0.smithi079.stdout:Sep 21 12:36:04 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-0[34281]: 2024-09-21T12:36:04.020+0000 7f31cd558640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T12:36:04.442 INFO:journalctl@ceph.osd.0.smithi079.stdout:Sep 21 12:36:04 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-0[34281]: 2024-09-21T12:36:04.020+0000 7f31cd558640 -1 osd.0 111 *** Got signal Terminated *** 2024-09-21T12:36:04.442 INFO:journalctl@ceph.osd.0.smithi079.stdout:Sep 21 12:36:04 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-0[34281]: 2024-09-21T12:36:04.020+0000 7f31cd558640 -1 osd.0 111 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T12:36:09.942 INFO:journalctl@ceph.osd.0.smithi079.stdout:Sep 21 12:36:09 smithi079 podman[80555]: 2024-09-21 12:36:09.641133752 +0000 UTC m=+5.733338561 container died 46b0cbf47ea9d46cd31924d41d3a9a4f7beb27484bd8d489de16fbe5159e31f7 (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-33168022-7814-11ef-baf6-efdc52797490-osd-0, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, RELEASE=reef-6a1bdf8, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=) 2024-09-21T12:36:10.286 INFO:journalctl@ceph.osd.0.smithi079.stdout:Sep 21 12:36:10 smithi079 podman[80555]: 2024-09-21 12:36:10.034439566 +0000 UTC m=+6.126644377 container cleanup 46b0cbf47ea9d46cd31924d41d3a9a4f7beb27484bd8d489de16fbe5159e31f7 (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-33168022-7814-11ef-baf6-efdc52797490-osd-0, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=reef-6a1bdf8) 2024-09-21T12:36:10.286 INFO:journalctl@ceph.osd.0.smithi079.stdout:Sep 21 12:36:10 smithi079 bash[80555]: ceph-33168022-7814-11ef-baf6-efdc52797490-osd-0 2024-09-21T12:36:10.691 INFO:journalctl@ceph.osd.0.smithi079.stdout:Sep 21 12:36:10 smithi079 podman[80829]: 2024-09-21 12:36:10.285705424 +0000 UTC m=+0.654167055 container remove 46b0cbf47ea9d46cd31924d41d3a9a4f7beb27484bd8d489de16fbe5159e31f7 (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-33168022-7814-11ef-baf6-efdc52797490-osd-0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=reef-6a1bdf8, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-09-21T12:36:11.082 INFO:journalctl@ceph.osd.0.smithi079.stdout:Sep 21 12:36:10 smithi079 podman[80926]: 2024-09-21 12:36:10.680844062 +0000 UTC m=+0.016161310 image pull 0adab276c97e6f3143ffbca5eeff37fcd7dc3b5c0f339a9f747381d9993602de quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6 2024-09-21T12:36:11.083 INFO:journalctl@ceph.osd.0.smithi079.stdout:Sep 21 12:36:10 smithi079 podman[80926]: 2024-09-21 12:36:10.795785794 +0000 UTC m=+0.131103084 container create 48184c2e1e580191e4d8eb7dc734c13b26517885c3d84c014506897aef889525 (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-33168022-7814-11ef-baf6-efdc52797490-osd-0-deactivate, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, RELEASE=reef-6a1bdf8, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-09-21T12:36:11.083 INFO:journalctl@ceph.osd.0.smithi079.stdout:Sep 21 12:36:11 smithi079 podman[80926]: 2024-09-21 12:36:11.081845442 +0000 UTC m=+0.417162699 container init 48184c2e1e580191e4d8eb7dc734c13b26517885c3d84c014506897aef889525 (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-33168022-7814-11ef-baf6-efdc52797490-osd-0-deactivate, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, ceph=True, RELEASE=reef-6a1bdf8, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-09-21T12:36:11.442 INFO:journalctl@ceph.osd.0.smithi079.stdout:Sep 21 12:36:11 smithi079 podman[80926]: 2024-09-21 12:36:11.085187478 +0000 UTC m=+0.420504727 container start 48184c2e1e580191e4d8eb7dc734c13b26517885c3d84c014506897aef889525 (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-33168022-7814-11ef-baf6-efdc52797490-osd-0-deactivate, io.buildah.version=1.37.2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=reef-6a1bdf8, GIT_CLEAN=True, org.label-schema.build-date=20240731) 2024-09-21T12:36:11.442 INFO:journalctl@ceph.osd.0.smithi079.stdout:Sep 21 12:36:11 smithi079 podman[80926]: 2024-09-21 12:36:11.157018509 +0000 UTC m=+0.492335762 container attach 48184c2e1e580191e4d8eb7dc734c13b26517885c3d84c014506897aef889525 (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-33168022-7814-11ef-baf6-efdc52797490-osd-0-deactivate, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=reef-6a1bdf8, ceph=True) 2024-09-21T12:36:11.442 INFO:journalctl@ceph.osd.0.smithi079.stdout:Sep 21 12:36:11 smithi079 podman[80926]: 2024-09-21 12:36:11.265521802 +0000 UTC m=+0.600839050 container died 48184c2e1e580191e4d8eb7dc734c13b26517885c3d84c014506897aef889525 (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-33168022-7814-11ef-baf6-efdc52797490-osd-0-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=reef-6a1bdf8, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-09-21T12:36:11.827 DEBUG:teuthology.orchestra.run.smithi079:> sudo pkill -f 'journalctl -f -n 0 -u ceph-33168022-7814-11ef-baf6-efdc52797490@osd.0.service' 2024-09-21T12:36:11.869 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T12:36:11.869 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-09-21T12:36:11.869 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-09-21T12:36:11.869 DEBUG:teuthology.orchestra.run.smithi079:> sudo systemctl stop ceph-33168022-7814-11ef-baf6-efdc52797490@osd.1 2024-09-21T12:36:12.192 INFO:journalctl@ceph.osd.1.smithi079.stdout:Sep 21 12:36:11 smithi079 systemd[1]: Stopping Ceph osd.1 for 33168022-7814-11ef-baf6-efdc52797490... 2024-09-21T12:36:12.691 INFO:journalctl@ceph.osd.1.smithi079.stdout:Sep 21 12:36:12 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-1[37753]: 2024-09-21T12:36:12.196+0000 7f26a72c3640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T12:36:12.692 INFO:journalctl@ceph.osd.1.smithi079.stdout:Sep 21 12:36:12 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-1[37753]: 2024-09-21T12:36:12.196+0000 7f26a72c3640 -1 osd.1 111 *** Got signal Terminated *** 2024-09-21T12:36:12.692 INFO:journalctl@ceph.osd.1.smithi079.stdout:Sep 21 12:36:12 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-1[37753]: 2024-09-21T12:36:12.196+0000 7f26a72c3640 -1 osd.1 111 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T12:36:17.797 INFO:journalctl@ceph.osd.1.smithi079.stdout:Sep 21 12:36:17 smithi079 podman[81028]: 2024-09-21 12:36:17.539044244 +0000 UTC m=+5.448665178 container died 6f56f34ef26f1f644781fe152af3610f1631f607fc9425ff52c4ac633116085f (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-33168022-7814-11ef-baf6-efdc52797490-osd-1, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=reef-6a1bdf8, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-09-21T12:36:18.173 INFO:journalctl@ceph.osd.1.smithi079.stdout:Sep 21 12:36:17 smithi079 podman[81028]: 2024-09-21 12:36:17.970751563 +0000 UTC m=+5.880372490 container cleanup 6f56f34ef26f1f644781fe152af3610f1631f607fc9425ff52c4ac633116085f (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-33168022-7814-11ef-baf6-efdc52797490-osd-1, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=reef-6a1bdf8, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True) 2024-09-21T12:36:18.173 INFO:journalctl@ceph.osd.1.smithi079.stdout:Sep 21 12:36:17 smithi079 bash[81028]: ceph-33168022-7814-11ef-baf6-efdc52797490-osd-1 2024-09-21T12:36:18.442 INFO:journalctl@ceph.osd.1.smithi079.stdout:Sep 21 12:36:18 smithi079 podman[81675]: 2024-09-21 12:36:18.17242143 +0000 UTC m=+0.633574687 container remove 6f56f34ef26f1f644781fe152af3610f1631f607fc9425ff52c4ac633116085f (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-33168022-7814-11ef-baf6-efdc52797490-osd-1, org.label-schema.build-date=20240731, RELEASE=reef-6a1bdf8, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T12:36:18.843 INFO:journalctl@ceph.osd.1.smithi079.stdout:Sep 21 12:36:18 smithi079 podman[81762]: 2024-09-21 12:36:18.479945495 +0000 UTC m=+0.015814620 image pull 0adab276c97e6f3143ffbca5eeff37fcd7dc3b5c0f339a9f747381d9993602de quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6 2024-09-21T12:36:18.843 INFO:journalctl@ceph.osd.1.smithi079.stdout:Sep 21 12:36:18 smithi079 podman[81762]: 2024-09-21 12:36:18.606459941 +0000 UTC m=+0.142329053 container create 4690ef589dd64d1d4b3856bf00a0ca1d15a76d970d3ff84f48c4143f7d967c97 (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-33168022-7814-11ef-baf6-efdc52797490-osd-1-deactivate, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=reef-6a1bdf8, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T12:36:19.192 INFO:journalctl@ceph.osd.1.smithi079.stdout:Sep 21 12:36:18 smithi079 podman[81762]: 2024-09-21 12:36:18.842540952 +0000 UTC m=+0.378410071 container init 4690ef589dd64d1d4b3856bf00a0ca1d15a76d970d3ff84f48c4143f7d967c97 (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-33168022-7814-11ef-baf6-efdc52797490-osd-1-deactivate, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , RELEASE=reef-6a1bdf8, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731) 2024-09-21T12:36:19.192 INFO:journalctl@ceph.osd.1.smithi079.stdout:Sep 21 12:36:18 smithi079 podman[81762]: 2024-09-21 12:36:18.845891326 +0000 UTC m=+0.381760440 container start 4690ef589dd64d1d4b3856bf00a0ca1d15a76d970d3ff84f48c4143f7d967c97 (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-33168022-7814-11ef-baf6-efdc52797490-osd-1-deactivate, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=reef-6a1bdf8, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T12:36:19.192 INFO:journalctl@ceph.osd.1.smithi079.stdout:Sep 21 12:36:18 smithi079 podman[81762]: 2024-09-21 12:36:18.917426605 +0000 UTC m=+0.453295736 container attach 4690ef589dd64d1d4b3856bf00a0ca1d15a76d970d3ff84f48c4143f7d967c97 (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-33168022-7814-11ef-baf6-efdc52797490-osd-1-deactivate, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=reef-6a1bdf8, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-09-21T12:36:19.192 INFO:journalctl@ceph.osd.1.smithi079.stdout:Sep 21 12:36:19 smithi079 podman[81762]: 2024-09-21 12:36:19.036718441 +0000 UTC m=+0.572587575 container died 4690ef589dd64d1d4b3856bf00a0ca1d15a76d970d3ff84f48c4143f7d967c97 (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-33168022-7814-11ef-baf6-efdc52797490-osd-1-deactivate, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-6a1bdf8, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T12:36:19.574 INFO:journalctl@ceph.osd.1.smithi079.stdout:Sep 21 12:36:19 smithi079 podman[81785]: 2024-09-21 12:36:19.562774065 +0000 UTC m=+0.516484555 container remove 4690ef589dd64d1d4b3856bf00a0ca1d15a76d970d3ff84f48c4143f7d967c97 (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-33168022-7814-11ef-baf6-efdc52797490-osd-1-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, RELEASE=reef-6a1bdf8, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T12:36:19.580 DEBUG:teuthology.orchestra.run.smithi079:> sudo pkill -f 'journalctl -f -n 0 -u ceph-33168022-7814-11ef-baf6-efdc52797490@osd.1.service' 2024-09-21T12:36:19.620 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T12:36:19.620 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-09-21T12:36:19.620 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-09-21T12:36:19.620 DEBUG:teuthology.orchestra.run.smithi079:> sudo systemctl stop ceph-33168022-7814-11ef-baf6-efdc52797490@osd.2 2024-09-21T12:36:19.942 INFO:journalctl@ceph.osd.2.smithi079.stdout:Sep 21 12:36:19 smithi079 systemd[1]: Stopping Ceph osd.2 for 33168022-7814-11ef-baf6-efdc52797490... 2024-09-21T12:36:20.441 INFO:journalctl@ceph.osd.2.smithi079.stdout:Sep 21 12:36:19 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-2[41222]: 2024-09-21T12:36:19.967+0000 7fb3d4c3f640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T12:36:20.442 INFO:journalctl@ceph.osd.2.smithi079.stdout:Sep 21 12:36:19 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-2[41222]: 2024-09-21T12:36:19.967+0000 7fb3d4c3f640 -1 osd.2 111 *** Got signal Terminated *** 2024-09-21T12:36:20.442 INFO:journalctl@ceph.osd.2.smithi079.stdout:Sep 21 12:36:19 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-2[41222]: 2024-09-21T12:36:19.967+0000 7fb3d4c3f640 -1 osd.2 111 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T12:36:22.177 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:36:21 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[63610]: ts=2024-09-21T12:36:21.728Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph-exporter msg="Unable to refresh target groups" err="Get \"http://172.21.15.104:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.104:8765: connect: connection refused" 2024-09-21T12:36:22.177 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:36:21 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[63610]: ts=2024-09-21T12:36:21.731Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=node msg="Unable to refresh target groups" err="Get \"http://172.21.15.104:8765/sd/prometheus/sd-config?service=node-exporter\": dial tcp 172.21.15.104:8765: connect: connection refused" 2024-09-21T12:36:22.178 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:36:21 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[63610]: ts=2024-09-21T12:36:21.731Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=nfs msg="Unable to refresh target groups" err="Get \"http://172.21.15.104:8765/sd/prometheus/sd-config?service=nfs\": dial tcp 172.21.15.104:8765: connect: connection refused" 2024-09-21T12:36:22.178 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:36:21 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[63610]: ts=2024-09-21T12:36:21.731Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=nvmeof msg="Unable to refresh target groups" err="Get \"http://172.21.15.104:8765/sd/prometheus/sd-config?service=nvmeof\": dial tcp 172.21.15.104:8765: connect: connection refused" 2024-09-21T12:36:22.178 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:36:21 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[63610]: ts=2024-09-21T12:36:21.733Z caller=refresh.go:99 level=error component="discovery manager notify" discovery=http config=config-0 msg="Unable to refresh target groups" err="Get \"http://172.21.15.104:8765/sd/prometheus/sd-config?service=alertmanager\": dial tcp 172.21.15.104:8765: connect: connection refused" 2024-09-21T12:36:22.178 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:36:21 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[63610]: ts=2024-09-21T12:36:21.734Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=smb msg="Unable to refresh target groups" err="Get \"http://172.21.15.104:8765/sd/prometheus/sd-config?service=smb\": dial tcp 172.21.15.104:8765: connect: connection refused" 2024-09-21T12:36:22.178 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:36:21 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[63610]: ts=2024-09-21T12:36:21.734Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph msg="Unable to refresh target groups" err="Get \"http://172.21.15.104:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.104:8765: connect: connection refused" 2024-09-21T12:36:25.692 INFO:journalctl@ceph.osd.2.smithi079.stdout:Sep 21 12:36:25 smithi079 podman[81864]: 2024-09-21 12:36:25.514242748 +0000 UTC m=+5.665847235 container died 9a4767c655787d149e4fdf6db7c5f1a2139196b366ecbb05b5af9c8aa6043a4e (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-33168022-7814-11ef-baf6-efdc52797490-osd-2, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-6a1bdf8, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-09-21T12:36:26.070 INFO:journalctl@ceph.osd.2.smithi079.stdout:Sep 21 12:36:25 smithi079 podman[81864]: 2024-09-21 12:36:25.867504726 +0000 UTC m=+6.019109248 container cleanup 9a4767c655787d149e4fdf6db7c5f1a2139196b366ecbb05b5af9c8aa6043a4e (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-33168022-7814-11ef-baf6-efdc52797490-osd-2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, RELEASE=reef-6a1bdf8, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-09-21T12:36:26.070 INFO:journalctl@ceph.osd.2.smithi079.stdout:Sep 21 12:36:25 smithi079 bash[81864]: ceph-33168022-7814-11ef-baf6-efdc52797490-osd-2 2024-09-21T12:36:26.329 INFO:journalctl@ceph.osd.2.smithi079.stdout:Sep 21 12:36:26 smithi079 podman[81893]: 2024-09-21 12:36:26.068907148 +0000 UTC m=+0.682309276 container remove 9a4767c655787d149e4fdf6db7c5f1a2139196b366ecbb05b5af9c8aa6043a4e (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-33168022-7814-11ef-baf6-efdc52797490-osd-2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=reef-6a1bdf8, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, ceph=True, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2) 2024-09-21T12:36:27.942 INFO:journalctl@ceph.osd.2.smithi079.stdout:Sep 21 12:36:27 smithi079 podman[82154]: 2024-09-21 12:36:27.586803491 +0000 UTC m=+0.017140697 image pull 0adab276c97e6f3143ffbca5eeff37fcd7dc3b5c0f339a9f747381d9993602de quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6 2024-09-21T12:36:27.942 INFO:journalctl@ceph.osd.2.smithi079.stdout:Sep 21 12:36:27 smithi079 podman[82154]: 2024-09-21 12:36:27.7196071 +0000 UTC m=+0.149944286 container create 1aaae1c2c3f48d2ff935351dacdf2e51779939331eff87384b61e71b565b760e (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-33168022-7814-11ef-baf6-efdc52797490-osd-2-deactivate, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=reef-6a1bdf8, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T12:36:28.682 INFO:journalctl@ceph.osd.2.smithi079.stdout:Sep 21 12:36:28 smithi079 podman[82154]: 2024-09-21 12:36:28.476208664 +0000 UTC m=+0.906545867 container init 1aaae1c2c3f48d2ff935351dacdf2e51779939331eff87384b61e71b565b760e (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-33168022-7814-11ef-baf6-efdc52797490-osd-2-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, RELEASE=reef-6a1bdf8, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux ) 2024-09-21T12:36:28.683 INFO:journalctl@ceph.osd.2.smithi079.stdout:Sep 21 12:36:28 smithi079 podman[82154]: 2024-09-21 12:36:28.47957998 +0000 UTC m=+0.909917167 container start 1aaae1c2c3f48d2ff935351dacdf2e51779939331eff87384b61e71b565b760e (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-33168022-7814-11ef-baf6-efdc52797490-osd-2-deactivate, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, RELEASE=reef-6a1bdf8, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T12:36:28.683 INFO:journalctl@ceph.osd.2.smithi079.stdout:Sep 21 12:36:28 smithi079 podman[82154]: 2024-09-21 12:36:28.5676741 +0000 UTC m=+0.998011300 container attach 1aaae1c2c3f48d2ff935351dacdf2e51779939331eff87384b61e71b565b760e (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-33168022-7814-11ef-baf6-efdc52797490-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , RELEASE=reef-6a1bdf8) 2024-09-21T12:36:28.683 INFO:journalctl@ceph.osd.2.smithi079.stdout:Sep 21 12:36:28 smithi079 podman[82154]: 2024-09-21 12:36:28.68186297 +0000 UTC m=+1.112200166 container died 1aaae1c2c3f48d2ff935351dacdf2e51779939331eff87384b61e71b565b760e (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-33168022-7814-11ef-baf6-efdc52797490-osd-2-deactivate, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, RELEASE=reef-6a1bdf8, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-09-21T12:36:29.284 DEBUG:teuthology.orchestra.run.smithi079:> sudo pkill -f 'journalctl -f -n 0 -u ceph-33168022-7814-11ef-baf6-efdc52797490@osd.2.service' 2024-09-21T12:36:29.321 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T12:36:29.322 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-09-21T12:36:29.322 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-09-21T12:36:29.322 DEBUG:teuthology.orchestra.run.smithi079:> sudo systemctl stop ceph-33168022-7814-11ef-baf6-efdc52797490@osd.3 2024-09-21T12:36:29.691 INFO:journalctl@ceph.osd.3.smithi079.stdout:Sep 21 12:36:29 smithi079 systemd[1]: Stopping Ceph osd.3 for 33168022-7814-11ef-baf6-efdc52797490... 2024-09-21T12:36:30.191 INFO:journalctl@ceph.osd.3.smithi079.stdout:Sep 21 12:36:29 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-3[44685]: 2024-09-21T12:36:29.690+0000 7f483a77e640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T12:36:30.192 INFO:journalctl@ceph.osd.3.smithi079.stdout:Sep 21 12:36:29 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-3[44685]: 2024-09-21T12:36:29.690+0000 7f483a77e640 -1 osd.3 111 *** Got signal Terminated *** 2024-09-21T12:36:30.192 INFO:journalctl@ceph.osd.3.smithi079.stdout:Sep 21 12:36:29 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-3[44685]: 2024-09-21T12:36:29.690+0000 7f483a77e640 -1 osd.3 111 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T12:36:31.677 INFO:journalctl@ceph.osd.4.smithi104.stdout:Sep 21 12:36:31 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-4[31795]: 2024-09-21T12:36:31.301+0000 7fd29e4f8640 -1 osd.4 111 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-09-21T12:36:05.823630+0000 front 2024-09-21T12:36:05.823736+0000 (oldest deadline 2024-09-21T12:36:30.523599+0000) 2024-09-21T12:36:32.288 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 12:36:32 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-7[42127]: 2024-09-21T12:36:32.025+0000 7f552c86a640 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-09-21T12:36:07.830828+0000 front 2024-09-21T12:36:07.830777+0000 (oldest deadline 2024-09-21T12:36:31.231331+0000) 2024-09-21T12:36:32.676 INFO:journalctl@ceph.osd.4.smithi104.stdout:Sep 21 12:36:32 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-4[31795]: 2024-09-21T12:36:32.286+0000 7fd29e4f8640 -1 osd.4 111 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-09-21T12:36:05.823630+0000 front 2024-09-21T12:36:05.823736+0000 (oldest deadline 2024-09-21T12:36:30.523599+0000) 2024-09-21T12:36:33.192 INFO:journalctl@ceph.osd.3.smithi079.stdout:Sep 21 12:36:32 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-3[44685]: 2024-09-21T12:36:32.892+0000 7f4836d92640 -1 osd.3 111 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-09-21T12:36:08.467130+0000 front 2024-09-21T12:36:08.467250+0000 (oldest deadline 2024-09-21T12:36:31.967253+0000) 2024-09-21T12:36:33.302 INFO:journalctl@ceph.osd.4.smithi104.stdout:Sep 21 12:36:33 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-4[31795]: 2024-09-21T12:36:33.266+0000 7fd29e4f8640 -1 osd.4 111 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-09-21T12:36:05.823630+0000 front 2024-09-21T12:36:05.823736+0000 (oldest deadline 2024-09-21T12:36:30.523599+0000) 2024-09-21T12:36:33.302 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 12:36:33 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-7[42127]: 2024-09-21T12:36:33.046+0000 7f552c86a640 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-09-21T12:36:07.830828+0000 front 2024-09-21T12:36:07.830777+0000 (oldest deadline 2024-09-21T12:36:31.231331+0000) 2024-09-21T12:36:34.177 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 12:36:34 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-7[42127]: 2024-09-21T12:36:34.051+0000 7f552c86a640 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-09-21T12:36:07.830828+0000 front 2024-09-21T12:36:07.830777+0000 (oldest deadline 2024-09-21T12:36:31.231331+0000) 2024-09-21T12:36:34.177 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 12:36:33 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-6[38758]: 2024-09-21T12:36:33.913+0000 7f19815f0640 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-09-21T12:36:08.568374+0000 front 2024-09-21T12:36:08.568570+0000 (oldest deadline 2024-09-21T12:36:33.268485+0000) 2024-09-21T12:36:34.441 INFO:journalctl@ceph.osd.3.smithi079.stdout:Sep 21 12:36:33 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-3[44685]: 2024-09-21T12:36:33.942+0000 7f4836d92640 -1 osd.3 111 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-09-21T12:36:08.467130+0000 front 2024-09-21T12:36:08.467250+0000 (oldest deadline 2024-09-21T12:36:31.967253+0000) 2024-09-21T12:36:34.677 INFO:journalctl@ceph.osd.4.smithi104.stdout:Sep 21 12:36:34 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-4[31795]: 2024-09-21T12:36:34.288+0000 7fd29e4f8640 -1 osd.4 111 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-09-21T12:36:05.823630+0000 front 2024-09-21T12:36:05.823736+0000 (oldest deadline 2024-09-21T12:36:30.523599+0000) 2024-09-21T12:36:35.061 INFO:journalctl@ceph.osd.5.smithi104.stdout:Sep 21 12:36:34 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-5[35230]: 2024-09-21T12:36:34.754+0000 7f8151cd6640 -1 osd.5 111 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-09-21T12:36:08.590631+0000 front 2024-09-21T12:36:08.590548+0000 (oldest deadline 2024-09-21T12:36:33.890612+0000) 2024-09-21T12:36:35.062 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 12:36:34 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-6[38758]: 2024-09-21T12:36:34.956+0000 7f19815f0640 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-09-21T12:36:08.568374+0000 front 2024-09-21T12:36:08.568570+0000 (oldest deadline 2024-09-21T12:36:33.268485+0000) 2024-09-21T12:36:35.427 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 12:36:35 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-7[42127]: 2024-09-21T12:36:35.058+0000 7f552c86a640 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-09-21T12:36:07.830828+0000 front 2024-09-21T12:36:07.830777+0000 (oldest deadline 2024-09-21T12:36:31.231331+0000) 2024-09-21T12:36:35.427 INFO:journalctl@ceph.osd.4.smithi104.stdout:Sep 21 12:36:35 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-4[31795]: 2024-09-21T12:36:35.265+0000 7fd29e4f8640 -1 osd.4 111 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-09-21T12:36:05.823630+0000 front 2024-09-21T12:36:05.823736+0000 (oldest deadline 2024-09-21T12:36:30.523599+0000) 2024-09-21T12:36:35.440 INFO:journalctl@ceph.osd.3.smithi079.stdout:Sep 21 12:36:35 smithi079 podman[82365]: 2024-09-21 12:36:35.141798894 +0000 UTC m=+5.569927179 container died 1e7af3834ad3cd9cb20c888ca435f910114711e95738e034aafac665d4245c73 (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-33168022-7814-11ef-baf6-efdc52797490-osd-3, GIT_CLEAN=True, io.buildah.version=1.37.2, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-6a1bdf8, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-09-21T12:36:35.692 INFO:journalctl@ceph.osd.3.smithi079.stdout:Sep 21 12:36:35 smithi079 podman[82365]: 2024-09-21 12:36:35.588048503 +0000 UTC m=+6.016176823 container cleanup 1e7af3834ad3cd9cb20c888ca435f910114711e95738e034aafac665d4245c73 (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-33168022-7814-11ef-baf6-efdc52797490-osd-3, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=reef-6a1bdf8, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T12:36:35.692 INFO:journalctl@ceph.osd.3.smithi079.stdout:Sep 21 12:36:35 smithi079 bash[82365]: ceph-33168022-7814-11ef-baf6-efdc52797490-osd-3 2024-09-21T12:36:36.079 INFO:journalctl@ceph.osd.5.smithi104.stdout:Sep 21 12:36:35 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-5[35230]: 2024-09-21T12:36:35.796+0000 7f8151cd6640 -1 osd.5 111 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-09-21T12:36:08.590631+0000 front 2024-09-21T12:36:08.590548+0000 (oldest deadline 2024-09-21T12:36:33.890612+0000) 2024-09-21T12:36:36.079 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 12:36:35 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-6[38758]: 2024-09-21T12:36:35.959+0000 7f19815f0640 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-09-21T12:36:08.568374+0000 front 2024-09-21T12:36:08.568570+0000 (oldest deadline 2024-09-21T12:36:33.268485+0000) 2024-09-21T12:36:36.192 INFO:journalctl@ceph.osd.3.smithi079.stdout:Sep 21 12:36:35 smithi079 podman[82852]: 2024-09-21 12:36:35.764561072 +0000 UTC m=+0.613554966 container remove 1e7af3834ad3cd9cb20c888ca435f910114711e95738e034aafac665d4245c73 (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-33168022-7814-11ef-baf6-efdc52797490-osd-3, GIT_BRANCH=HEAD, RELEASE=reef-6a1bdf8, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T12:36:36.427 INFO:journalctl@ceph.osd.4.smithi104.stdout:Sep 21 12:36:36 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-4[31795]: 2024-09-21T12:36:36.233+0000 7fd29e4f8640 -1 osd.4 111 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-09-21T12:36:05.823630+0000 front 2024-09-21T12:36:05.823736+0000 (oldest deadline 2024-09-21T12:36:30.523599+0000) 2024-09-21T12:36:36.427 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 12:36:36 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-7[42127]: 2024-09-21T12:36:36.077+0000 7f552c86a640 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-09-21T12:36:07.830828+0000 front 2024-09-21T12:36:07.830777+0000 (oldest deadline 2024-09-21T12:36:31.231331+0000) 2024-09-21T12:36:36.477 INFO:journalctl@ceph.osd.3.smithi079.stdout:Sep 21 12:36:36 smithi079 podman[82939]: 2024-09-21 12:36:36.198537955 +0000 UTC m=+0.112568821 container create 4be8b4c46214b9a602ec6a45525fe6dfd00084fe862191fb66cd0a01775ec3fa (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-33168022-7814-11ef-baf6-efdc52797490-osd-3-deactivate, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=reef-6a1bdf8, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T12:36:36.477 INFO:journalctl@ceph.osd.3.smithi079.stdout:Sep 21 12:36:36 smithi079 podman[82939]: 2024-09-21 12:36:36.101779266 +0000 UTC m=+0.015810149 image pull 0adab276c97e6f3143ffbca5eeff37fcd7dc3b5c0f339a9f747381d9993602de quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6 2024-09-21T12:36:36.942 INFO:journalctl@ceph.osd.3.smithi079.stdout:Sep 21 12:36:36 smithi079 podman[82939]: 2024-09-21 12:36:36.476362905 +0000 UTC m=+0.390393781 container init 4be8b4c46214b9a602ec6a45525fe6dfd00084fe862191fb66cd0a01775ec3fa (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-33168022-7814-11ef-baf6-efdc52797490-osd-3-deactivate, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=reef-6a1bdf8, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-09-21T12:36:36.942 INFO:journalctl@ceph.osd.3.smithi079.stdout:Sep 21 12:36:36 smithi079 podman[82939]: 2024-09-21 12:36:36.479895613 +0000 UTC m=+0.393926481 container start 4be8b4c46214b9a602ec6a45525fe6dfd00084fe862191fb66cd0a01775ec3fa (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-33168022-7814-11ef-baf6-efdc52797490-osd-3-deactivate, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=reef-6a1bdf8, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T12:36:36.942 INFO:journalctl@ceph.osd.3.smithi079.stdout:Sep 21 12:36:36 smithi079 podman[82939]: 2024-09-21 12:36:36.559303966 +0000 UTC m=+0.473334849 container attach 4be8b4c46214b9a602ec6a45525fe6dfd00084fe862191fb66cd0a01775ec3fa (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-33168022-7814-11ef-baf6-efdc52797490-osd-3-deactivate, ceph=True, org.label-schema.vendor=CentOS, RELEASE=reef-6a1bdf8, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0) 2024-09-21T12:36:36.942 INFO:journalctl@ceph.osd.3.smithi079.stdout:Sep 21 12:36:36 smithi079 podman[82939]: 2024-09-21 12:36:36.672107646 +0000 UTC m=+0.586138514 container died 4be8b4c46214b9a602ec6a45525fe6dfd00084fe862191fb66cd0a01775ec3fa (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-33168022-7814-11ef-baf6-efdc52797490-osd-3-deactivate, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, RELEASE=reef-6a1bdf8, io.buildah.version=1.37.2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T12:36:37.051 INFO:journalctl@ceph.osd.5.smithi104.stdout:Sep 21 12:36:36 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-5[35230]: 2024-09-21T12:36:36.771+0000 7f8151cd6640 -1 osd.5 111 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-09-21T12:36:08.590631+0000 front 2024-09-21T12:36:08.590548+0000 (oldest deadline 2024-09-21T12:36:33.890612+0000) 2024-09-21T12:36:37.052 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 12:36:36 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-6[38758]: 2024-09-21T12:36:36.908+0000 7f19815f0640 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-09-21T12:36:08.568374+0000 front 2024-09-21T12:36:08.568570+0000 (oldest deadline 2024-09-21T12:36:33.268485+0000) 2024-09-21T12:36:37.196 DEBUG:teuthology.orchestra.run.smithi079:> sudo pkill -f 'journalctl -f -n 0 -u ceph-33168022-7814-11ef-baf6-efdc52797490@osd.3.service' 2024-09-21T12:36:37.220 INFO:journalctl@ceph.osd.3.smithi079.stdout:Sep 21 12:36:37 smithi079 podman[82962]: 2024-09-21 12:36:37.171182538 +0000 UTC m=+0.493074745 container remove 4be8b4c46214b9a602ec6a45525fe6dfd00084fe862191fb66cd0a01775ec3fa (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-33168022-7814-11ef-baf6-efdc52797490-osd-3-deactivate, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=reef-6a1bdf8, maintainer=Guillaume Abrioux ) 2024-09-21T12:36:37.221 INFO:journalctl@ceph.osd.3.smithi079.stdout:Sep 21 12:36:37 smithi079 systemd[1]: ceph-33168022-7814-11ef-baf6-efdc52797490@osd.3.service: Deactivated successfully. 2024-09-21T12:36:37.221 INFO:journalctl@ceph.osd.3.smithi079.stdout:Sep 21 12:36:37 smithi079 systemd[1]: Stopped Ceph osd.3 for 33168022-7814-11ef-baf6-efdc52797490. 2024-09-21T12:36:37.221 INFO:journalctl@ceph.osd.3.smithi079.stdout:Sep 21 12:36:37 smithi079 systemd[1]: ceph-33168022-7814-11ef-baf6-efdc52797490@osd.3.service: Consumed 5.983s CPU time. 2024-09-21T12:36:37.236 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T12:36:37.236 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-09-21T12:36:37.236 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-09-21T12:36:37.236 DEBUG:teuthology.orchestra.run.smithi104:> sudo systemctl stop ceph-33168022-7814-11ef-baf6-efdc52797490@osd.4 2024-09-21T12:36:37.426 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 12:36:37 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-7[42127]: 2024-09-21T12:36:37.049+0000 7f552c86a640 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-09-21T12:36:07.830828+0000 front 2024-09-21T12:36:07.830777+0000 (oldest deadline 2024-09-21T12:36:31.231331+0000) 2024-09-21T12:36:37.427 INFO:journalctl@ceph.osd.4.smithi104.stdout:Sep 21 12:36:37 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-4[31795]: 2024-09-21T12:36:37.255+0000 7fd29e4f8640 -1 osd.4 111 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-09-21T12:36:05.823630+0000 front 2024-09-21T12:36:05.823736+0000 (oldest deadline 2024-09-21T12:36:30.523599+0000) 2024-09-21T12:36:37.427 INFO:journalctl@ceph.osd.4.smithi104.stdout:Sep 21 12:36:37 smithi104 systemd[1]: Stopping Ceph osd.4 for 33168022-7814-11ef-baf6-efdc52797490... 2024-09-21T12:36:37.821 INFO:journalctl@ceph.osd.4.smithi104.stdout:Sep 21 12:36:37 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-4[31795]: 2024-09-21T12:36:37.522+0000 7fd2a5eec640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T12:36:37.822 INFO:journalctl@ceph.osd.4.smithi104.stdout:Sep 21 12:36:37 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-4[31795]: 2024-09-21T12:36:37.522+0000 7fd2a5eec640 -1 osd.4 111 *** Got signal Terminated *** 2024-09-21T12:36:37.822 INFO:journalctl@ceph.osd.4.smithi104.stdout:Sep 21 12:36:37 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-4[31795]: 2024-09-21T12:36:37.522+0000 7fd2a5eec640 -1 osd.4 111 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T12:36:38.083 INFO:journalctl@ceph.osd.5.smithi104.stdout:Sep 21 12:36:37 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-5[35230]: 2024-09-21T12:36:37.819+0000 7f8151cd6640 -1 osd.5 111 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-09-21T12:36:08.590631+0000 front 2024-09-21T12:36:08.590548+0000 (oldest deadline 2024-09-21T12:36:33.890612+0000) 2024-09-21T12:36:38.083 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 12:36:37 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-6[38758]: 2024-09-21T12:36:37.933+0000 7f19815f0640 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-09-21T12:36:08.568374+0000 front 2024-09-21T12:36:08.568570+0000 (oldest deadline 2024-09-21T12:36:33.268485+0000) 2024-09-21T12:36:38.426 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 12:36:38 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-7[42127]: 2024-09-21T12:36:38.081+0000 7f552c86a640 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-09-21T12:36:07.830828+0000 front 2024-09-21T12:36:07.830777+0000 (oldest deadline 2024-09-21T12:36:31.231331+0000) 2024-09-21T12:36:38.427 INFO:journalctl@ceph.osd.4.smithi104.stdout:Sep 21 12:36:38 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-4[31795]: 2024-09-21T12:36:38.236+0000 7fd29e4f8640 -1 osd.4 111 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-09-21T12:36:05.823630+0000 front 2024-09-21T12:36:05.823736+0000 (oldest deadline 2024-09-21T12:36:30.523599+0000) 2024-09-21T12:36:39.122 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 12:36:38 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-6[38758]: 2024-09-21T12:36:38.894+0000 7f19815f0640 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-09-21T12:36:08.568374+0000 front 2024-09-21T12:36:08.568570+0000 (oldest deadline 2024-09-21T12:36:33.268485+0000) 2024-09-21T12:36:39.122 INFO:journalctl@ceph.osd.5.smithi104.stdout:Sep 21 12:36:38 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-5[35230]: 2024-09-21T12:36:38.865+0000 7f8151cd6640 -1 osd.5 111 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-09-21T12:36:08.590631+0000 front 2024-09-21T12:36:08.590548+0000 (oldest deadline 2024-09-21T12:36:33.890612+0000) 2024-09-21T12:36:39.122 INFO:journalctl@ceph.osd.5.smithi104.stdout:Sep 21 12:36:38 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-5[35230]: 2024-09-21T12:36:38.865+0000 7f8151cd6640 -1 osd.5 111 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2024-09-21T12:36:13.891098+0000 front 2024-09-21T12:36:13.891110+0000 (oldest deadline 2024-09-21T12:36:38.591096+0000) 2024-09-21T12:36:39.426 INFO:journalctl@ceph.osd.4.smithi104.stdout:Sep 21 12:36:39 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-4[31795]: 2024-09-21T12:36:39.246+0000 7fd29e4f8640 -1 osd.4 111 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-09-21T12:36:05.823630+0000 front 2024-09-21T12:36:05.823736+0000 (oldest deadline 2024-09-21T12:36:30.523599+0000) 2024-09-21T12:36:39.427 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 12:36:39 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-7[42127]: 2024-09-21T12:36:39.120+0000 7f552c86a640 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-09-21T12:36:07.830828+0000 front 2024-09-21T12:36:07.830777+0000 (oldest deadline 2024-09-21T12:36:31.231331+0000) 2024-09-21T12:36:40.178 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 12:36:39 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-6[38758]: 2024-09-21T12:36:39.856+0000 7f19815f0640 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-09-21T12:36:08.568374+0000 front 2024-09-21T12:36:08.568570+0000 (oldest deadline 2024-09-21T12:36:33.268485+0000) 2024-09-21T12:36:40.178 INFO:journalctl@ceph.osd.5.smithi104.stdout:Sep 21 12:36:39 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-5[35230]: 2024-09-21T12:36:39.857+0000 7f8151cd6640 -1 osd.5 111 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-09-21T12:36:08.590631+0000 front 2024-09-21T12:36:08.590548+0000 (oldest deadline 2024-09-21T12:36:33.890612+0000) 2024-09-21T12:36:40.178 INFO:journalctl@ceph.osd.5.smithi104.stdout:Sep 21 12:36:39 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-5[35230]: 2024-09-21T12:36:39.857+0000 7f8151cd6640 -1 osd.5 111 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2024-09-21T12:36:13.891098+0000 front 2024-09-21T12:36:13.891110+0000 (oldest deadline 2024-09-21T12:36:38.591096+0000) 2024-09-21T12:36:40.178 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 12:36:40 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-7[42127]: 2024-09-21T12:36:40.072+0000 7f552c86a640 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-09-21T12:36:07.830828+0000 front 2024-09-21T12:36:07.830777+0000 (oldest deadline 2024-09-21T12:36:31.231331+0000) 2024-09-21T12:36:40.491 INFO:journalctl@ceph.osd.4.smithi104.stdout:Sep 21 12:36:40 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-4[31795]: 2024-09-21T12:36:40.215+0000 7fd29e4f8640 -1 osd.4 111 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-09-21T12:36:05.823630+0000 front 2024-09-21T12:36:05.823736+0000 (oldest deadline 2024-09-21T12:36:30.523599+0000) 2024-09-21T12:36:41.176 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 12:36:41 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-7[42127]: 2024-09-21T12:36:41.073+0000 7f552c86a640 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-09-21T12:36:07.830828+0000 front 2024-09-21T12:36:07.830777+0000 (oldest deadline 2024-09-21T12:36:31.231331+0000) 2024-09-21T12:36:41.177 INFO:journalctl@ceph.osd.5.smithi104.stdout:Sep 21 12:36:40 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-5[35230]: 2024-09-21T12:36:40.878+0000 7f8151cd6640 -1 osd.5 111 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-09-21T12:36:08.590631+0000 front 2024-09-21T12:36:08.590548+0000 (oldest deadline 2024-09-21T12:36:33.890612+0000) 2024-09-21T12:36:41.177 INFO:journalctl@ceph.osd.5.smithi104.stdout:Sep 21 12:36:40 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-5[35230]: 2024-09-21T12:36:40.878+0000 7f8151cd6640 -1 osd.5 111 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2024-09-21T12:36:13.891098+0000 front 2024-09-21T12:36:13.891110+0000 (oldest deadline 2024-09-21T12:36:38.591096+0000) 2024-09-21T12:36:41.177 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 12:36:40 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-6[38758]: 2024-09-21T12:36:40.870+0000 7f19815f0640 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-09-21T12:36:08.568374+0000 front 2024-09-21T12:36:08.568570+0000 (oldest deadline 2024-09-21T12:36:33.268485+0000) 2024-09-21T12:36:41.177 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 12:36:40 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-6[38758]: 2024-09-21T12:36:40.870+0000 7f19815f0640 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2024-09-21T12:36:16.069952+0000 front 2024-09-21T12:36:16.069931+0000 (oldest deadline 2024-09-21T12:36:40.769786+0000) 2024-09-21T12:36:41.613 INFO:journalctl@ceph.osd.4.smithi104.stdout:Sep 21 12:36:41 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-4[31795]: 2024-09-21T12:36:41.254+0000 7fd29e4f8640 -1 osd.4 111 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-09-21T12:36:05.823630+0000 front 2024-09-21T12:36:05.823736+0000 (oldest deadline 2024-09-21T12:36:30.523599+0000) 2024-09-21T12:36:42.177 INFO:journalctl@ceph.osd.5.smithi104.stdout:Sep 21 12:36:41 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-5[35230]: 2024-09-21T12:36:41.890+0000 7f8151cd6640 -1 osd.5 111 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-09-21T12:36:08.590631+0000 front 2024-09-21T12:36:08.590548+0000 (oldest deadline 2024-09-21T12:36:33.890612+0000) 2024-09-21T12:36:42.177 INFO:journalctl@ceph.osd.5.smithi104.stdout:Sep 21 12:36:41 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-5[35230]: 2024-09-21T12:36:41.890+0000 7f8151cd6640 -1 osd.5 111 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2024-09-21T12:36:13.891098+0000 front 2024-09-21T12:36:13.891110+0000 (oldest deadline 2024-09-21T12:36:38.591096+0000) 2024-09-21T12:36:42.177 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 12:36:41 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-6[38758]: 2024-09-21T12:36:41.911+0000 7f19815f0640 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-09-21T12:36:08.568374+0000 front 2024-09-21T12:36:08.568570+0000 (oldest deadline 2024-09-21T12:36:33.268485+0000) 2024-09-21T12:36:42.177 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 12:36:41 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-6[38758]: 2024-09-21T12:36:41.911+0000 7f19815f0640 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2024-09-21T12:36:16.069952+0000 front 2024-09-21T12:36:16.069931+0000 (oldest deadline 2024-09-21T12:36:40.769786+0000) 2024-09-21T12:36:42.178 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 12:36:42 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-7[42127]: 2024-09-21T12:36:42.114+0000 7f552c86a640 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-09-21T12:36:07.830828+0000 front 2024-09-21T12:36:07.830777+0000 (oldest deadline 2024-09-21T12:36:31.231331+0000) 2024-09-21T12:36:42.178 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 12:36:42 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-7[42127]: 2024-09-21T12:36:42.114+0000 7f552c86a640 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2024-09-21T12:36:17.132004+0000 front 2024-09-21T12:36:17.131993+0000 (oldest deadline 2024-09-21T12:36:41.232131+0000) 2024-09-21T12:36:42.676 INFO:journalctl@ceph.osd.4.smithi104.stdout:Sep 21 12:36:42 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-4[31795]: 2024-09-21T12:36:42.223+0000 7fd29e4f8640 -1 osd.4 111 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-09-21T12:36:05.823630+0000 front 2024-09-21T12:36:05.823736+0000 (oldest deadline 2024-09-21T12:36:30.523599+0000) 2024-09-21T12:36:42.677 INFO:journalctl@ceph.osd.4.smithi104.stdout:Sep 21 12:36:42 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-4[31795]: 2024-09-21T12:36:42.223+0000 7fd29e4f8640 -1 osd.4 111 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2024-09-21T12:36:15.724820+0000 front 2024-09-21T12:36:15.724697+0000 (oldest deadline 2024-09-21T12:36:42.125348+0000) 2024-09-21T12:36:42.995 INFO:journalctl@ceph.osd.5.smithi104.stdout:Sep 21 12:36:42 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-5[35230]: 2024-09-21T12:36:42.900+0000 7f8151cd6640 -1 osd.5 111 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-09-21T12:36:08.590631+0000 front 2024-09-21T12:36:08.590548+0000 (oldest deadline 2024-09-21T12:36:33.890612+0000) 2024-09-21T12:36:42.995 INFO:journalctl@ceph.osd.5.smithi104.stdout:Sep 21 12:36:42 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-5[35230]: 2024-09-21T12:36:42.900+0000 7f8151cd6640 -1 osd.5 111 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2024-09-21T12:36:13.891098+0000 front 2024-09-21T12:36:13.891110+0000 (oldest deadline 2024-09-21T12:36:38.591096+0000) 2024-09-21T12:36:42.996 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 12:36:42 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-6[38758]: 2024-09-21T12:36:42.905+0000 7f19815f0640 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-09-21T12:36:08.568374+0000 front 2024-09-21T12:36:08.568570+0000 (oldest deadline 2024-09-21T12:36:33.268485+0000) 2024-09-21T12:36:42.996 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 12:36:42 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-6[38758]: 2024-09-21T12:36:42.905+0000 7f19815f0640 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2024-09-21T12:36:16.069952+0000 front 2024-09-21T12:36:16.069931+0000 (oldest deadline 2024-09-21T12:36:40.769786+0000) 2024-09-21T12:36:43.338 INFO:journalctl@ceph.osd.4.smithi104.stdout:Sep 21 12:36:42 smithi104 podman[65854]: 2024-09-21 12:36:42.994465378 +0000 UTC m=+5.562213519 container died 2b027321f96132ae2769d8262024af47bef9b30dd820c642364f9221731f3cb7 (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-33168022-7814-11ef-baf6-efdc52797490-osd-4, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=reef-6a1bdf8, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T12:36:43.338 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 12:36:43 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-7[42127]: 2024-09-21T12:36:43.118+0000 7f552c86a640 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-09-21T12:36:07.830828+0000 front 2024-09-21T12:36:07.830777+0000 (oldest deadline 2024-09-21T12:36:31.231331+0000) 2024-09-21T12:36:43.338 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 12:36:43 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-7[42127]: 2024-09-21T12:36:43.118+0000 7f552c86a640 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2024-09-21T12:36:17.132004+0000 front 2024-09-21T12:36:17.131993+0000 (oldest deadline 2024-09-21T12:36:41.232131+0000) 2024-09-21T12:36:43.622 INFO:journalctl@ceph.osd.4.smithi104.stdout:Sep 21 12:36:43 smithi104 podman[65854]: 2024-09-21 12:36:43.427728438 +0000 UTC m=+5.995476578 container cleanup 2b027321f96132ae2769d8262024af47bef9b30dd820c642364f9221731f3cb7 (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-33168022-7814-11ef-baf6-efdc52797490-osd-4, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=reef-6a1bdf8, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240731) 2024-09-21T12:36:43.622 INFO:journalctl@ceph.osd.4.smithi104.stdout:Sep 21 12:36:43 smithi104 bash[65854]: ceph-33168022-7814-11ef-baf6-efdc52797490-osd-4 2024-09-21T12:36:43.914 INFO:journalctl@ceph.osd.4.smithi104.stdout:Sep 21 12:36:43 smithi104 podman[65978]: 2024-09-21 12:36:43.620938885 +0000 UTC m=+0.627685670 container remove 2b027321f96132ae2769d8262024af47bef9b30dd820c642364f9221731f3cb7 (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-33168022-7814-11ef-baf6-efdc52797490-osd-4, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, RELEASE=reef-6a1bdf8, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS) 2024-09-21T12:36:44.166 INFO:journalctl@ceph.osd.4.smithi104.stdout:Sep 21 12:36:44 smithi104 podman[66148]: 2024-09-21 12:36:44.030251379 +0000 UTC m=+0.100963928 container create 712953f3a148d5eba54f3b55c60127c014de4119bef8cbbe716d7d8c3742079a (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-33168022-7814-11ef-baf6-efdc52797490-osd-4-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=reef-6a1bdf8, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS) 2024-09-21T12:36:44.166 INFO:journalctl@ceph.osd.4.smithi104.stdout:Sep 21 12:36:44 smithi104 podman[66148]: 2024-09-21 12:36:43.946565925 +0000 UTC m=+0.017278432 image pull 0adab276c97e6f3143ffbca5eeff37fcd7dc3b5c0f339a9f747381d9993602de quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6 2024-09-21T12:36:44.166 INFO:journalctl@ceph.osd.5.smithi104.stdout:Sep 21 12:36:43 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-5[35230]: 2024-09-21T12:36:43.935+0000 7f8151cd6640 -1 osd.5 111 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-09-21T12:36:08.590631+0000 front 2024-09-21T12:36:08.590548+0000 (oldest deadline 2024-09-21T12:36:33.890612+0000) 2024-09-21T12:36:44.166 INFO:journalctl@ceph.osd.5.smithi104.stdout:Sep 21 12:36:43 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-5[35230]: 2024-09-21T12:36:43.935+0000 7f8151cd6640 -1 osd.5 111 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2024-09-21T12:36:13.891098+0000 front 2024-09-21T12:36:13.891110+0000 (oldest deadline 2024-09-21T12:36:38.591096+0000) 2024-09-21T12:36:44.167 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 12:36:43 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-6[38758]: 2024-09-21T12:36:43.944+0000 7f19815f0640 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-09-21T12:36:08.568374+0000 front 2024-09-21T12:36:08.568570+0000 (oldest deadline 2024-09-21T12:36:33.268485+0000) 2024-09-21T12:36:44.167 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 12:36:43 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-6[38758]: 2024-09-21T12:36:43.944+0000 7f19815f0640 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2024-09-21T12:36:16.069952+0000 front 2024-09-21T12:36:16.069931+0000 (oldest deadline 2024-09-21T12:36:40.769786+0000) 2024-09-21T12:36:44.167 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 12:36:44 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-7[42127]: 2024-09-21T12:36:44.076+0000 7f552c86a640 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-09-21T12:36:07.830828+0000 front 2024-09-21T12:36:07.830777+0000 (oldest deadline 2024-09-21T12:36:31.231331+0000) 2024-09-21T12:36:44.167 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 12:36:44 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-7[42127]: 2024-09-21T12:36:44.076+0000 7f552c86a640 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2024-09-21T12:36:17.132004+0000 front 2024-09-21T12:36:17.131993+0000 (oldest deadline 2024-09-21T12:36:41.232131+0000) 2024-09-21T12:36:44.676 INFO:journalctl@ceph.osd.4.smithi104.stdout:Sep 21 12:36:44 smithi104 podman[66148]: 2024-09-21 12:36:44.258086251 +0000 UTC m=+0.328798760 container init 712953f3a148d5eba54f3b55c60127c014de4119bef8cbbe716d7d8c3742079a (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-33168022-7814-11ef-baf6-efdc52797490-osd-4-deactivate, RELEASE=reef-6a1bdf8, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T12:36:44.677 INFO:journalctl@ceph.osd.4.smithi104.stdout:Sep 21 12:36:44 smithi104 podman[66148]: 2024-09-21 12:36:44.261547233 +0000 UTC m=+0.332259745 container start 712953f3a148d5eba54f3b55c60127c014de4119bef8cbbe716d7d8c3742079a (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-33168022-7814-11ef-baf6-efdc52797490-osd-4-deactivate, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-6a1bdf8, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-09-21T12:36:44.677 INFO:journalctl@ceph.osd.4.smithi104.stdout:Sep 21 12:36:44 smithi104 podman[66148]: 2024-09-21 12:36:44.332371529 +0000 UTC m=+0.403084050 container attach 712953f3a148d5eba54f3b55c60127c014de4119bef8cbbe716d7d8c3742079a (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-33168022-7814-11ef-baf6-efdc52797490-osd-4-deactivate, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=reef-6a1bdf8, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True) 2024-09-21T12:36:44.677 INFO:journalctl@ceph.osd.4.smithi104.stdout:Sep 21 12:36:44 smithi104 podman[66148]: 2024-09-21 12:36:44.46218282 +0000 UTC m=+0.532895333 container died 712953f3a148d5eba54f3b55c60127c014de4119bef8cbbe716d7d8c3742079a (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-33168022-7814-11ef-baf6-efdc52797490-osd-4-deactivate, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-6a1bdf8, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-09-21T12:36:45.031 INFO:journalctl@ceph.osd.4.smithi104.stdout:Sep 21 12:36:45 smithi104 podman[66235]: 2024-09-21 12:36:45.027377911 +0000 UTC m=+0.559153245 container remove 712953f3a148d5eba54f3b55c60127c014de4119bef8cbbe716d7d8c3742079a (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-33168022-7814-11ef-baf6-efdc52797490-osd-4-deactivate, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=reef-6a1bdf8, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True) 2024-09-21T12:36:45.031 INFO:journalctl@ceph.osd.5.smithi104.stdout:Sep 21 12:36:44 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-5[35230]: 2024-09-21T12:36:44.984+0000 7f8151cd6640 -1 osd.5 111 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-09-21T12:36:08.590631+0000 front 2024-09-21T12:36:08.590548+0000 (oldest deadline 2024-09-21T12:36:33.890612+0000) 2024-09-21T12:36:45.031 INFO:journalctl@ceph.osd.5.smithi104.stdout:Sep 21 12:36:44 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-5[35230]: 2024-09-21T12:36:44.984+0000 7f8151cd6640 -1 osd.5 111 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2024-09-21T12:36:13.891098+0000 front 2024-09-21T12:36:13.891110+0000 (oldest deadline 2024-09-21T12:36:38.591096+0000) 2024-09-21T12:36:45.032 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 12:36:44 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-6[38758]: 2024-09-21T12:36:44.937+0000 7f19815f0640 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-09-21T12:36:08.568374+0000 front 2024-09-21T12:36:08.568570+0000 (oldest deadline 2024-09-21T12:36:33.268485+0000) 2024-09-21T12:36:45.032 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 12:36:44 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-6[38758]: 2024-09-21T12:36:44.937+0000 7f19815f0640 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2024-09-21T12:36:16.069952+0000 front 2024-09-21T12:36:16.069931+0000 (oldest deadline 2024-09-21T12:36:40.769786+0000) 2024-09-21T12:36:45.050 DEBUG:teuthology.orchestra.run.smithi104:> sudo pkill -f 'journalctl -f -n 0 -u ceph-33168022-7814-11ef-baf6-efdc52797490@osd.4.service' 2024-09-21T12:36:45.090 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T12:36:45.090 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-09-21T12:36:45.090 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-09-21T12:36:45.090 DEBUG:teuthology.orchestra.run.smithi104:> sudo systemctl stop ceph-33168022-7814-11ef-baf6-efdc52797490@osd.5 2024-09-21T12:36:45.423 INFO:journalctl@ceph.osd.5.smithi104.stdout:Sep 21 12:36:45 smithi104 systemd[1]: Stopping Ceph osd.5 for 33168022-7814-11ef-baf6-efdc52797490... 2024-09-21T12:36:45.423 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 12:36:45 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-7[42127]: 2024-09-21T12:36:45.055+0000 7f552c86a640 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-09-21T12:36:07.830828+0000 front 2024-09-21T12:36:07.830777+0000 (oldest deadline 2024-09-21T12:36:31.231331+0000) 2024-09-21T12:36:45.423 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 12:36:45 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-7[42127]: 2024-09-21T12:36:45.055+0000 7f552c86a640 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2024-09-21T12:36:17.132004+0000 front 2024-09-21T12:36:17.131993+0000 (oldest deadline 2024-09-21T12:36:41.232131+0000) 2024-09-21T12:36:45.676 INFO:journalctl@ceph.osd.5.smithi104.stdout:Sep 21 12:36:45 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-5[35230]: 2024-09-21T12:36:45.420+0000 7f81596ca640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T12:36:45.677 INFO:journalctl@ceph.osd.5.smithi104.stdout:Sep 21 12:36:45 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-5[35230]: 2024-09-21T12:36:45.420+0000 7f81596ca640 -1 osd.5 111 *** Got signal Terminated *** 2024-09-21T12:36:45.677 INFO:journalctl@ceph.osd.5.smithi104.stdout:Sep 21 12:36:45 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-5[35230]: 2024-09-21T12:36:45.420+0000 7f81596ca640 -1 osd.5 111 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T12:36:46.177 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 12:36:45 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-6[38758]: 2024-09-21T12:36:45.891+0000 7f19815f0640 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-09-21T12:36:08.568374+0000 front 2024-09-21T12:36:08.568570+0000 (oldest deadline 2024-09-21T12:36:33.268485+0000) 2024-09-21T12:36:46.177 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 12:36:45 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-6[38758]: 2024-09-21T12:36:45.891+0000 7f19815f0640 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2024-09-21T12:36:16.069952+0000 front 2024-09-21T12:36:16.069931+0000 (oldest deadline 2024-09-21T12:36:40.769786+0000) 2024-09-21T12:36:46.177 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 12:36:45 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-6[38758]: 2024-09-21T12:36:45.891+0000 7f19815f0640 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.79:6822 osd.2 since back 2024-09-21T12:36:20.770367+0000 front 2024-09-21T12:36:20.770384+0000 (oldest deadline 2024-09-21T12:36:45.470242+0000) 2024-09-21T12:36:46.177 INFO:journalctl@ceph.osd.5.smithi104.stdout:Sep 21 12:36:45 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-5[35230]: 2024-09-21T12:36:45.950+0000 7f8151cd6640 -1 osd.5 111 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-09-21T12:36:08.590631+0000 front 2024-09-21T12:36:08.590548+0000 (oldest deadline 2024-09-21T12:36:33.890612+0000) 2024-09-21T12:36:46.177 INFO:journalctl@ceph.osd.5.smithi104.stdout:Sep 21 12:36:45 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-5[35230]: 2024-09-21T12:36:45.950+0000 7f8151cd6640 -1 osd.5 111 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2024-09-21T12:36:13.891098+0000 front 2024-09-21T12:36:13.891110+0000 (oldest deadline 2024-09-21T12:36:38.591096+0000) 2024-09-21T12:36:46.178 INFO:journalctl@ceph.osd.5.smithi104.stdout:Sep 21 12:36:45 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-5[35230]: 2024-09-21T12:36:45.950+0000 7f8151cd6640 -1 osd.5 111 heartbeat_check: no reply from 172.21.15.79:6822 osd.2 since back 2024-09-21T12:36:23.891980+0000 front 2024-09-21T12:36:23.892035+0000 (oldest deadline 2024-09-21T12:36:45.591966+0000) 2024-09-21T12:36:46.178 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 12:36:46 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-7[42127]: 2024-09-21T12:36:46.017+0000 7f552c86a640 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-09-21T12:36:07.830828+0000 front 2024-09-21T12:36:07.830777+0000 (oldest deadline 2024-09-21T12:36:31.231331+0000) 2024-09-21T12:36:46.178 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 12:36:46 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-7[42127]: 2024-09-21T12:36:46.017+0000 7f552c86a640 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2024-09-21T12:36:17.132004+0000 front 2024-09-21T12:36:17.131993+0000 (oldest deadline 2024-09-21T12:36:41.232131+0000) 2024-09-21T12:36:46.178 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 12:36:46 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-7[42127]: 2024-09-21T12:36:46.017+0000 7f552c86a640 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.79:6822 osd.2 since back 2024-09-21T12:36:21.232698+0000 front 2024-09-21T12:36:21.232643+0000 (oldest deadline 2024-09-21T12:36:45.932532+0000) 2024-09-21T12:36:47.142 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 12:36:46 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-7[42127]: 2024-09-21T12:36:46.994+0000 7f552c86a640 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-09-21T12:36:07.830828+0000 front 2024-09-21T12:36:07.830777+0000 (oldest deadline 2024-09-21T12:36:31.231331+0000) 2024-09-21T12:36:47.142 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 12:36:46 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-7[42127]: 2024-09-21T12:36:46.994+0000 7f552c86a640 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2024-09-21T12:36:17.132004+0000 front 2024-09-21T12:36:17.131993+0000 (oldest deadline 2024-09-21T12:36:41.232131+0000) 2024-09-21T12:36:47.142 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 12:36:46 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-7[42127]: 2024-09-21T12:36:46.994+0000 7f552c86a640 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.79:6822 osd.2 since back 2024-09-21T12:36:21.232698+0000 front 2024-09-21T12:36:21.232643+0000 (oldest deadline 2024-09-21T12:36:45.932532+0000) 2024-09-21T12:36:47.143 INFO:journalctl@ceph.osd.5.smithi104.stdout:Sep 21 12:36:46 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-5[35230]: 2024-09-21T12:36:46.901+0000 7f8151cd6640 -1 osd.5 111 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-09-21T12:36:08.590631+0000 front 2024-09-21T12:36:08.590548+0000 (oldest deadline 2024-09-21T12:36:33.890612+0000) 2024-09-21T12:36:47.143 INFO:journalctl@ceph.osd.5.smithi104.stdout:Sep 21 12:36:46 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-5[35230]: 2024-09-21T12:36:46.901+0000 7f8151cd6640 -1 osd.5 111 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2024-09-21T12:36:13.891098+0000 front 2024-09-21T12:36:13.891110+0000 (oldest deadline 2024-09-21T12:36:38.591096+0000) 2024-09-21T12:36:47.143 INFO:journalctl@ceph.osd.5.smithi104.stdout:Sep 21 12:36:46 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-5[35230]: 2024-09-21T12:36:46.901+0000 7f8151cd6640 -1 osd.5 111 heartbeat_check: no reply from 172.21.15.79:6822 osd.2 since back 2024-09-21T12:36:23.891980+0000 front 2024-09-21T12:36:23.892035+0000 (oldest deadline 2024-09-21T12:36:45.591966+0000) 2024-09-21T12:36:47.143 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 12:36:46 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-6[38758]: 2024-09-21T12:36:46.871+0000 7f19815f0640 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-09-21T12:36:08.568374+0000 front 2024-09-21T12:36:08.568570+0000 (oldest deadline 2024-09-21T12:36:33.268485+0000) 2024-09-21T12:36:47.143 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 12:36:46 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-6[38758]: 2024-09-21T12:36:46.871+0000 7f19815f0640 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2024-09-21T12:36:16.069952+0000 front 2024-09-21T12:36:16.069931+0000 (oldest deadline 2024-09-21T12:36:40.769786+0000) 2024-09-21T12:36:47.144 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 12:36:46 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-6[38758]: 2024-09-21T12:36:46.871+0000 7f19815f0640 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.79:6822 osd.2 since back 2024-09-21T12:36:20.770367+0000 front 2024-09-21T12:36:20.770384+0000 (oldest deadline 2024-09-21T12:36:45.470242+0000) 2024-09-21T12:36:48.177 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 12:36:47 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-6[38758]: 2024-09-21T12:36:47.829+0000 7f19815f0640 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-09-21T12:36:08.568374+0000 front 2024-09-21T12:36:08.568570+0000 (oldest deadline 2024-09-21T12:36:33.268485+0000) 2024-09-21T12:36:48.177 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 12:36:47 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-6[38758]: 2024-09-21T12:36:47.829+0000 7f19815f0640 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2024-09-21T12:36:16.069952+0000 front 2024-09-21T12:36:16.069931+0000 (oldest deadline 2024-09-21T12:36:40.769786+0000) 2024-09-21T12:36:48.177 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 12:36:47 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-6[38758]: 2024-09-21T12:36:47.829+0000 7f19815f0640 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.79:6822 osd.2 since back 2024-09-21T12:36:20.770367+0000 front 2024-09-21T12:36:20.770384+0000 (oldest deadline 2024-09-21T12:36:45.470242+0000) 2024-09-21T12:36:48.177 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 12:36:47 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-7[42127]: 2024-09-21T12:36:47.947+0000 7f552c86a640 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-09-21T12:36:07.830828+0000 front 2024-09-21T12:36:07.830777+0000 (oldest deadline 2024-09-21T12:36:31.231331+0000) 2024-09-21T12:36:48.178 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 12:36:47 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-7[42127]: 2024-09-21T12:36:47.947+0000 7f552c86a640 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2024-09-21T12:36:17.132004+0000 front 2024-09-21T12:36:17.131993+0000 (oldest deadline 2024-09-21T12:36:41.232131+0000) 2024-09-21T12:36:48.178 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 12:36:47 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-7[42127]: 2024-09-21T12:36:47.947+0000 7f552c86a640 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.79:6822 osd.2 since back 2024-09-21T12:36:21.232698+0000 front 2024-09-21T12:36:21.232643+0000 (oldest deadline 2024-09-21T12:36:45.932532+0000) 2024-09-21T12:36:48.178 INFO:journalctl@ceph.osd.5.smithi104.stdout:Sep 21 12:36:47 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-5[35230]: 2024-09-21T12:36:47.899+0000 7f8151cd6640 -1 osd.5 111 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-09-21T12:36:08.590631+0000 front 2024-09-21T12:36:08.590548+0000 (oldest deadline 2024-09-21T12:36:33.890612+0000) 2024-09-21T12:36:48.178 INFO:journalctl@ceph.osd.5.smithi104.stdout:Sep 21 12:36:47 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-5[35230]: 2024-09-21T12:36:47.899+0000 7f8151cd6640 -1 osd.5 111 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2024-09-21T12:36:13.891098+0000 front 2024-09-21T12:36:13.891110+0000 (oldest deadline 2024-09-21T12:36:38.591096+0000) 2024-09-21T12:36:48.178 INFO:journalctl@ceph.osd.5.smithi104.stdout:Sep 21 12:36:47 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-5[35230]: 2024-09-21T12:36:47.899+0000 7f8151cd6640 -1 osd.5 111 heartbeat_check: no reply from 172.21.15.79:6822 osd.2 since back 2024-09-21T12:36:23.891980+0000 front 2024-09-21T12:36:23.892035+0000 (oldest deadline 2024-09-21T12:36:45.591966+0000) 2024-09-21T12:36:49.177 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 12:36:48 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-6[38758]: 2024-09-21T12:36:48.792+0000 7f19815f0640 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-09-21T12:36:08.568374+0000 front 2024-09-21T12:36:08.568570+0000 (oldest deadline 2024-09-21T12:36:33.268485+0000) 2024-09-21T12:36:49.177 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 12:36:48 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-6[38758]: 2024-09-21T12:36:48.792+0000 7f19815f0640 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2024-09-21T12:36:16.069952+0000 front 2024-09-21T12:36:16.069931+0000 (oldest deadline 2024-09-21T12:36:40.769786+0000) 2024-09-21T12:36:49.177 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 12:36:48 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-6[38758]: 2024-09-21T12:36:48.792+0000 7f19815f0640 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.79:6822 osd.2 since back 2024-09-21T12:36:20.770367+0000 front 2024-09-21T12:36:20.770384+0000 (oldest deadline 2024-09-21T12:36:45.470242+0000) 2024-09-21T12:36:49.178 INFO:journalctl@ceph.osd.5.smithi104.stdout:Sep 21 12:36:48 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-5[35230]: 2024-09-21T12:36:48.870+0000 7f8151cd6640 -1 osd.5 111 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-09-21T12:36:08.590631+0000 front 2024-09-21T12:36:08.590548+0000 (oldest deadline 2024-09-21T12:36:33.890612+0000) 2024-09-21T12:36:49.178 INFO:journalctl@ceph.osd.5.smithi104.stdout:Sep 21 12:36:48 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-5[35230]: 2024-09-21T12:36:48.870+0000 7f8151cd6640 -1 osd.5 111 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2024-09-21T12:36:13.891098+0000 front 2024-09-21T12:36:13.891110+0000 (oldest deadline 2024-09-21T12:36:38.591096+0000) 2024-09-21T12:36:49.178 INFO:journalctl@ceph.osd.5.smithi104.stdout:Sep 21 12:36:48 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-5[35230]: 2024-09-21T12:36:48.870+0000 7f8151cd6640 -1 osd.5 111 heartbeat_check: no reply from 172.21.15.79:6822 osd.2 since back 2024-09-21T12:36:23.891980+0000 front 2024-09-21T12:36:23.892035+0000 (oldest deadline 2024-09-21T12:36:45.591966+0000) 2024-09-21T12:36:49.179 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 12:36:48 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-7[42127]: 2024-09-21T12:36:48.925+0000 7f552c86a640 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-09-21T12:36:07.830828+0000 front 2024-09-21T12:36:07.830777+0000 (oldest deadline 2024-09-21T12:36:31.231331+0000) 2024-09-21T12:36:49.179 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 12:36:48 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-7[42127]: 2024-09-21T12:36:48.925+0000 7f552c86a640 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2024-09-21T12:36:17.132004+0000 front 2024-09-21T12:36:17.131993+0000 (oldest deadline 2024-09-21T12:36:41.232131+0000) 2024-09-21T12:36:49.179 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 12:36:48 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-7[42127]: 2024-09-21T12:36:48.925+0000 7f552c86a640 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.79:6822 osd.2 since back 2024-09-21T12:36:21.232698+0000 front 2024-09-21T12:36:21.232643+0000 (oldest deadline 2024-09-21T12:36:45.932532+0000) 2024-09-21T12:36:50.177 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 12:36:49 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-6[38758]: 2024-09-21T12:36:49.779+0000 7f19815f0640 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-09-21T12:36:08.568374+0000 front 2024-09-21T12:36:08.568570+0000 (oldest deadline 2024-09-21T12:36:33.268485+0000) 2024-09-21T12:36:50.177 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 12:36:49 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-6[38758]: 2024-09-21T12:36:49.779+0000 7f19815f0640 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2024-09-21T12:36:16.069952+0000 front 2024-09-21T12:36:16.069931+0000 (oldest deadline 2024-09-21T12:36:40.769786+0000) 2024-09-21T12:36:50.177 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 12:36:49 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-6[38758]: 2024-09-21T12:36:49.779+0000 7f19815f0640 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.79:6822 osd.2 since back 2024-09-21T12:36:20.770367+0000 front 2024-09-21T12:36:20.770384+0000 (oldest deadline 2024-09-21T12:36:45.470242+0000) 2024-09-21T12:36:50.177 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 12:36:49 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-7[42127]: 2024-09-21T12:36:49.926+0000 7f552c86a640 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-09-21T12:36:07.830828+0000 front 2024-09-21T12:36:07.830777+0000 (oldest deadline 2024-09-21T12:36:31.231331+0000) 2024-09-21T12:36:50.178 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 12:36:49 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-7[42127]: 2024-09-21T12:36:49.926+0000 7f552c86a640 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2024-09-21T12:36:17.132004+0000 front 2024-09-21T12:36:17.131993+0000 (oldest deadline 2024-09-21T12:36:41.232131+0000) 2024-09-21T12:36:50.178 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 12:36:49 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-7[42127]: 2024-09-21T12:36:49.926+0000 7f552c86a640 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.79:6822 osd.2 since back 2024-09-21T12:36:21.232698+0000 front 2024-09-21T12:36:21.232643+0000 (oldest deadline 2024-09-21T12:36:45.932532+0000) 2024-09-21T12:36:50.178 INFO:journalctl@ceph.osd.5.smithi104.stdout:Sep 21 12:36:49 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-5[35230]: 2024-09-21T12:36:49.867+0000 7f8151cd6640 -1 osd.5 111 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-09-21T12:36:08.590631+0000 front 2024-09-21T12:36:08.590548+0000 (oldest deadline 2024-09-21T12:36:33.890612+0000) 2024-09-21T12:36:50.178 INFO:journalctl@ceph.osd.5.smithi104.stdout:Sep 21 12:36:49 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-5[35230]: 2024-09-21T12:36:49.867+0000 7f8151cd6640 -1 osd.5 111 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2024-09-21T12:36:13.891098+0000 front 2024-09-21T12:36:13.891110+0000 (oldest deadline 2024-09-21T12:36:38.591096+0000) 2024-09-21T12:36:50.178 INFO:journalctl@ceph.osd.5.smithi104.stdout:Sep 21 12:36:49 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-5[35230]: 2024-09-21T12:36:49.867+0000 7f8151cd6640 -1 osd.5 111 heartbeat_check: no reply from 172.21.15.79:6822 osd.2 since back 2024-09-21T12:36:23.891980+0000 front 2024-09-21T12:36:23.892035+0000 (oldest deadline 2024-09-21T12:36:45.591966+0000) 2024-09-21T12:36:51.061 INFO:journalctl@ceph.osd.5.smithi104.stdout:Sep 21 12:36:50 smithi104 podman[66361]: 2024-09-21 12:36:50.763754033 +0000 UTC m=+5.433151548 container died 5e738fa75e97d512c579b3e0e785d79c732f1706d6073e6a2a2b9bf930f94544 (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-33168022-7814-11ef-baf6-efdc52797490-osd-5, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, RELEASE=reef-6a1bdf8, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-09-21T12:36:51.062 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 12:36:50 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-6[38758]: 2024-09-21T12:36:50.813+0000 7f19815f0640 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-09-21T12:36:08.568374+0000 front 2024-09-21T12:36:08.568570+0000 (oldest deadline 2024-09-21T12:36:33.268485+0000) 2024-09-21T12:36:51.062 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 12:36:50 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-6[38758]: 2024-09-21T12:36:50.813+0000 7f19815f0640 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2024-09-21T12:36:16.069952+0000 front 2024-09-21T12:36:16.069931+0000 (oldest deadline 2024-09-21T12:36:40.769786+0000) 2024-09-21T12:36:51.062 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 12:36:50 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-6[38758]: 2024-09-21T12:36:50.813+0000 7f19815f0640 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.79:6822 osd.2 since back 2024-09-21T12:36:20.770367+0000 front 2024-09-21T12:36:20.770384+0000 (oldest deadline 2024-09-21T12:36:45.470242+0000) 2024-09-21T12:36:51.062 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 12:36:50 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-7[42127]: 2024-09-21T12:36:50.926+0000 7f552c86a640 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-09-21T12:36:07.830828+0000 front 2024-09-21T12:36:07.830777+0000 (oldest deadline 2024-09-21T12:36:31.231331+0000) 2024-09-21T12:36:51.063 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 12:36:50 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-7[42127]: 2024-09-21T12:36:50.926+0000 7f552c86a640 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2024-09-21T12:36:17.132004+0000 front 2024-09-21T12:36:17.131993+0000 (oldest deadline 2024-09-21T12:36:41.232131+0000) 2024-09-21T12:36:51.063 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 12:36:50 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-7[42127]: 2024-09-21T12:36:50.926+0000 7f552c86a640 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.79:6822 osd.2 since back 2024-09-21T12:36:21.232698+0000 front 2024-09-21T12:36:21.232643+0000 (oldest deadline 2024-09-21T12:36:45.932532+0000) 2024-09-21T12:36:51.427 INFO:journalctl@ceph.osd.5.smithi104.stdout:Sep 21 12:36:51 smithi104 podman[66361]: 2024-09-21 12:36:51.259078056 +0000 UTC m=+5.928475613 container cleanup 5e738fa75e97d512c579b3e0e785d79c732f1706d6073e6a2a2b9bf930f94544 (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-33168022-7814-11ef-baf6-efdc52797490-osd-5, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, RELEASE=reef-6a1bdf8, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-09-21T12:36:51.427 INFO:journalctl@ceph.osd.5.smithi104.stdout:Sep 21 12:36:51 smithi104 bash[66361]: ceph-33168022-7814-11ef-baf6-efdc52797490-osd-5 2024-09-21T12:36:51.829 INFO:journalctl@ceph.osd.5.smithi104.stdout:Sep 21 12:36:51 smithi104 podman[66832]: 2024-09-21 12:36:51.518931282 +0000 UTC m=+0.751806837 container remove 5e738fa75e97d512c579b3e0e785d79c732f1706d6073e6a2a2b9bf930f94544 (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-33168022-7814-11ef-baf6-efdc52797490-osd-5, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=reef-6a1bdf8, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T12:36:52.106 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 12:36:51 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-7[42127]: 2024-09-21T12:36:51.899+0000 7f552c86a640 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-09-21T12:36:07.830828+0000 front 2024-09-21T12:36:07.830777+0000 (oldest deadline 2024-09-21T12:36:31.231331+0000) 2024-09-21T12:36:52.106 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 12:36:51 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-7[42127]: 2024-09-21T12:36:51.899+0000 7f552c86a640 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2024-09-21T12:36:17.132004+0000 front 2024-09-21T12:36:17.131993+0000 (oldest deadline 2024-09-21T12:36:41.232131+0000) 2024-09-21T12:36:52.106 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 12:36:51 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-7[42127]: 2024-09-21T12:36:51.899+0000 7f552c86a640 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.79:6822 osd.2 since back 2024-09-21T12:36:21.232698+0000 front 2024-09-21T12:36:21.232643+0000 (oldest deadline 2024-09-21T12:36:45.932532+0000) 2024-09-21T12:36:52.107 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 12:36:51 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-6[38758]: 2024-09-21T12:36:51.829+0000 7f19815f0640 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-09-21T12:36:08.568374+0000 front 2024-09-21T12:36:08.568570+0000 (oldest deadline 2024-09-21T12:36:33.268485+0000) 2024-09-21T12:36:52.107 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 12:36:51 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-6[38758]: 2024-09-21T12:36:51.829+0000 7f19815f0640 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2024-09-21T12:36:16.069952+0000 front 2024-09-21T12:36:16.069931+0000 (oldest deadline 2024-09-21T12:36:40.769786+0000) 2024-09-21T12:36:52.107 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 12:36:51 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-6[38758]: 2024-09-21T12:36:51.829+0000 7f19815f0640 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.79:6822 osd.2 since back 2024-09-21T12:36:20.770367+0000 front 2024-09-21T12:36:20.770384+0000 (oldest deadline 2024-09-21T12:36:45.470242+0000) 2024-09-21T12:36:52.107 INFO:journalctl@ceph.osd.5.smithi104.stdout:Sep 21 12:36:51 smithi104 podman[66920]: 2024-09-21 12:36:51.732224864 +0000 UTC m=+0.016389015 image pull 0adab276c97e6f3143ffbca5eeff37fcd7dc3b5c0f339a9f747381d9993602de quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6 2024-09-21T12:36:52.107 INFO:journalctl@ceph.osd.5.smithi104.stdout:Sep 21 12:36:51 smithi104 podman[66920]: 2024-09-21 12:36:51.836282839 +0000 UTC m=+0.120447035 container create 141e3dc6acbe4eef8d27ca7bd97e81b50db420b6af02dc4b6268f1807e6fbb9b (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-33168022-7814-11ef-baf6-efdc52797490-osd-5-deactivate, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=reef-6a1bdf8, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-09-21T12:36:52.107 INFO:journalctl@ceph.osd.5.smithi104.stdout:Sep 21 12:36:52 smithi104 podman[66920]: 2024-09-21 12:36:52.030403956 +0000 UTC m=+0.314568124 container init 141e3dc6acbe4eef8d27ca7bd97e81b50db420b6af02dc4b6268f1807e6fbb9b (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-33168022-7814-11ef-baf6-efdc52797490-osd-5-deactivate, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.37.2, RELEASE=reef-6a1bdf8, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-09-21T12:36:52.108 INFO:journalctl@ceph.osd.5.smithi104.stdout:Sep 21 12:36:52 smithi104 podman[66920]: 2024-09-21 12:36:52.033629131 +0000 UTC m=+0.317793281 container start 141e3dc6acbe4eef8d27ca7bd97e81b50db420b6af02dc4b6268f1807e6fbb9b (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-33168022-7814-11ef-baf6-efdc52797490-osd-5-deactivate, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, RELEASE=reef-6a1bdf8, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0) 2024-09-21T12:36:52.108 INFO:journalctl@ceph.osd.5.smithi104.stdout:Sep 21 12:36:52 smithi104 podman[66920]: 2024-09-21 12:36:52.105354023 +0000 UTC m=+0.389518197 container attach 141e3dc6acbe4eef8d27ca7bd97e81b50db420b6af02dc4b6268f1807e6fbb9b (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-33168022-7814-11ef-baf6-efdc52797490-osd-5-deactivate, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=reef-6a1bdf8, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T12:36:52.427 INFO:journalctl@ceph.osd.5.smithi104.stdout:Sep 21 12:36:52 smithi104 podman[66920]: 2024-09-21 12:36:52.222526075 +0000 UTC m=+0.506690236 container died 141e3dc6acbe4eef8d27ca7bd97e81b50db420b6af02dc4b6268f1807e6fbb9b (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-33168022-7814-11ef-baf6-efdc52797490-osd-5-deactivate, RELEASE=reef-6a1bdf8, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True) 2024-09-21T12:36:52.822 DEBUG:teuthology.orchestra.run.smithi104:> sudo pkill -f 'journalctl -f -n 0 -u ceph-33168022-7814-11ef-baf6-efdc52797490@osd.5.service' 2024-09-21T12:36:52.863 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T12:36:52.863 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-09-21T12:36:52.863 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-09-21T12:36:52.863 DEBUG:teuthology.orchestra.run.smithi104:> sudo systemctl stop ceph-33168022-7814-11ef-baf6-efdc52797490@osd.6 2024-09-21T12:36:53.176 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 12:36:52 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-7[42127]: 2024-09-21T12:36:52.887+0000 7f552c86a640 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-09-21T12:36:07.830828+0000 front 2024-09-21T12:36:07.830777+0000 (oldest deadline 2024-09-21T12:36:31.231331+0000) 2024-09-21T12:36:53.177 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 12:36:52 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-7[42127]: 2024-09-21T12:36:52.887+0000 7f552c86a640 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2024-09-21T12:36:17.132004+0000 front 2024-09-21T12:36:17.131993+0000 (oldest deadline 2024-09-21T12:36:41.232131+0000) 2024-09-21T12:36:53.177 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 12:36:52 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-7[42127]: 2024-09-21T12:36:52.887+0000 7f552c86a640 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.79:6822 osd.2 since back 2024-09-21T12:36:21.232698+0000 front 2024-09-21T12:36:21.232643+0000 (oldest deadline 2024-09-21T12:36:45.932532+0000) 2024-09-21T12:36:53.177 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 12:36:52 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-6[38758]: 2024-09-21T12:36:52.801+0000 7f19815f0640 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-09-21T12:36:08.568374+0000 front 2024-09-21T12:36:08.568570+0000 (oldest deadline 2024-09-21T12:36:33.268485+0000) 2024-09-21T12:36:53.177 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 12:36:52 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-6[38758]: 2024-09-21T12:36:52.801+0000 7f19815f0640 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2024-09-21T12:36:16.069952+0000 front 2024-09-21T12:36:16.069931+0000 (oldest deadline 2024-09-21T12:36:40.769786+0000) 2024-09-21T12:36:53.177 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 12:36:52 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-6[38758]: 2024-09-21T12:36:52.801+0000 7f19815f0640 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.79:6822 osd.2 since back 2024-09-21T12:36:20.770367+0000 front 2024-09-21T12:36:20.770384+0000 (oldest deadline 2024-09-21T12:36:45.470242+0000) 2024-09-21T12:36:53.177 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 12:36:52 smithi104 systemd[1]: Stopping Ceph osd.6 for 33168022-7814-11ef-baf6-efdc52797490... 2024-09-21T12:36:53.676 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 12:36:53 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-6[38758]: 2024-09-21T12:36:53.211+0000 7f1988fe4640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T12:36:53.677 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 12:36:53 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-6[38758]: 2024-09-21T12:36:53.211+0000 7f1988fe4640 -1 osd.6 111 *** Got signal Terminated *** 2024-09-21T12:36:53.677 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 12:36:53 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-6[38758]: 2024-09-21T12:36:53.211+0000 7f1988fe4640 -1 osd.6 111 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T12:36:54.176 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 12:36:53 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-6[38758]: 2024-09-21T12:36:53.786+0000 7f19815f0640 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-09-21T12:36:08.568374+0000 front 2024-09-21T12:36:08.568570+0000 (oldest deadline 2024-09-21T12:36:33.268485+0000) 2024-09-21T12:36:54.177 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 12:36:53 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-6[38758]: 2024-09-21T12:36:53.786+0000 7f19815f0640 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2024-09-21T12:36:16.069952+0000 front 2024-09-21T12:36:16.069931+0000 (oldest deadline 2024-09-21T12:36:40.769786+0000) 2024-09-21T12:36:54.177 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 12:36:53 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-6[38758]: 2024-09-21T12:36:53.786+0000 7f19815f0640 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.79:6822 osd.2 since back 2024-09-21T12:36:20.770367+0000 front 2024-09-21T12:36:20.770384+0000 (oldest deadline 2024-09-21T12:36:45.470242+0000) 2024-09-21T12:36:54.177 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 12:36:53 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-7[42127]: 2024-09-21T12:36:53.902+0000 7f552c86a640 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-09-21T12:36:07.830828+0000 front 2024-09-21T12:36:07.830777+0000 (oldest deadline 2024-09-21T12:36:31.231331+0000) 2024-09-21T12:36:54.177 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 12:36:53 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-7[42127]: 2024-09-21T12:36:53.902+0000 7f552c86a640 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2024-09-21T12:36:17.132004+0000 front 2024-09-21T12:36:17.131993+0000 (oldest deadline 2024-09-21T12:36:41.232131+0000) 2024-09-21T12:36:54.177 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 12:36:53 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-7[42127]: 2024-09-21T12:36:53.902+0000 7f552c86a640 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.79:6822 osd.2 since back 2024-09-21T12:36:21.232698+0000 front 2024-09-21T12:36:21.232643+0000 (oldest deadline 2024-09-21T12:36:45.932532+0000) 2024-09-21T12:36:55.177 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 12:36:54 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-6[38758]: 2024-09-21T12:36:54.762+0000 7f19815f0640 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-09-21T12:36:08.568374+0000 front 2024-09-21T12:36:08.568570+0000 (oldest deadline 2024-09-21T12:36:33.268485+0000) 2024-09-21T12:36:55.177 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 12:36:54 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-6[38758]: 2024-09-21T12:36:54.762+0000 7f19815f0640 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2024-09-21T12:36:16.069952+0000 front 2024-09-21T12:36:16.069931+0000 (oldest deadline 2024-09-21T12:36:40.769786+0000) 2024-09-21T12:36:55.177 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 12:36:54 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-6[38758]: 2024-09-21T12:36:54.762+0000 7f19815f0640 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.79:6822 osd.2 since back 2024-09-21T12:36:20.770367+0000 front 2024-09-21T12:36:20.770384+0000 (oldest deadline 2024-09-21T12:36:45.470242+0000) 2024-09-21T12:36:55.177 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 12:36:54 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-7[42127]: 2024-09-21T12:36:54.869+0000 7f552c86a640 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-09-21T12:36:07.830828+0000 front 2024-09-21T12:36:07.830777+0000 (oldest deadline 2024-09-21T12:36:31.231331+0000) 2024-09-21T12:36:55.177 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 12:36:54 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-7[42127]: 2024-09-21T12:36:54.869+0000 7f552c86a640 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2024-09-21T12:36:17.132004+0000 front 2024-09-21T12:36:17.131993+0000 (oldest deadline 2024-09-21T12:36:41.232131+0000) 2024-09-21T12:36:55.178 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 12:36:54 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-7[42127]: 2024-09-21T12:36:54.869+0000 7f552c86a640 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.79:6822 osd.2 since back 2024-09-21T12:36:21.232698+0000 front 2024-09-21T12:36:21.232643+0000 (oldest deadline 2024-09-21T12:36:45.932532+0000) 2024-09-21T12:36:56.176 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 12:36:55 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-6[38758]: 2024-09-21T12:36:55.757+0000 7f19815f0640 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-09-21T12:36:08.568374+0000 front 2024-09-21T12:36:08.568570+0000 (oldest deadline 2024-09-21T12:36:33.268485+0000) 2024-09-21T12:36:56.177 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 12:36:55 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-6[38758]: 2024-09-21T12:36:55.757+0000 7f19815f0640 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2024-09-21T12:36:16.069952+0000 front 2024-09-21T12:36:16.069931+0000 (oldest deadline 2024-09-21T12:36:40.769786+0000) 2024-09-21T12:36:56.177 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 12:36:55 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-6[38758]: 2024-09-21T12:36:55.757+0000 7f19815f0640 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.79:6822 osd.2 since back 2024-09-21T12:36:20.770367+0000 front 2024-09-21T12:36:20.770384+0000 (oldest deadline 2024-09-21T12:36:45.470242+0000) 2024-09-21T12:36:56.177 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 12:36:55 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-6[38758]: 2024-09-21T12:36:55.757+0000 7f19815f0640 -1 osd.6 111 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24526.0:928 4.1f 4:f95f44c2:::notify.0:head [watch ping cookie 94516886903552] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) 2024-09-21T12:36:56.177 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 12:36:55 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-7[42127]: 2024-09-21T12:36:55.834+0000 7f552c86a640 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-09-21T12:36:07.830828+0000 front 2024-09-21T12:36:07.830777+0000 (oldest deadline 2024-09-21T12:36:31.231331+0000) 2024-09-21T12:36:56.177 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 12:36:55 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-7[42127]: 2024-09-21T12:36:55.834+0000 7f552c86a640 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2024-09-21T12:36:17.132004+0000 front 2024-09-21T12:36:17.131993+0000 (oldest deadline 2024-09-21T12:36:41.232131+0000) 2024-09-21T12:36:56.178 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 12:36:55 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-7[42127]: 2024-09-21T12:36:55.834+0000 7f552c86a640 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.79:6822 osd.2 since back 2024-09-21T12:36:21.232698+0000 front 2024-09-21T12:36:21.232643+0000 (oldest deadline 2024-09-21T12:36:45.932532+0000) 2024-09-21T12:36:56.178 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 12:36:55 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-7[42127]: 2024-09-21T12:36:55.834+0000 7f552c86a640 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.79:6830 osd.3 since back 2024-09-21T12:36:30.633479+0000 front 2024-09-21T12:36:30.633439+0000 (oldest deadline 2024-09-21T12:36:55.333430+0000) 2024-09-21T12:36:57.177 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 12:36:56 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-6[38758]: 2024-09-21T12:36:56.721+0000 7f19815f0640 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-09-21T12:36:08.568374+0000 front 2024-09-21T12:36:08.568570+0000 (oldest deadline 2024-09-21T12:36:33.268485+0000) 2024-09-21T12:36:57.177 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 12:36:56 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-6[38758]: 2024-09-21T12:36:56.721+0000 7f19815f0640 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2024-09-21T12:36:16.069952+0000 front 2024-09-21T12:36:16.069931+0000 (oldest deadline 2024-09-21T12:36:40.769786+0000) 2024-09-21T12:36:57.177 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 12:36:56 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-6[38758]: 2024-09-21T12:36:56.721+0000 7f19815f0640 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.79:6822 osd.2 since back 2024-09-21T12:36:20.770367+0000 front 2024-09-21T12:36:20.770384+0000 (oldest deadline 2024-09-21T12:36:45.470242+0000) 2024-09-21T12:36:57.177 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 12:36:56 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-6[38758]: 2024-09-21T12:36:56.721+0000 7f19815f0640 -1 osd.6 111 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.24526.0:928 4.1f 4:f95f44c2:::notify.0:head [watch ping cookie 94516886903552] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) 2024-09-21T12:36:57.177 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 12:36:56 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-7[42127]: 2024-09-21T12:36:56.806+0000 7f552c86a640 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-09-21T12:36:07.830828+0000 front 2024-09-21T12:36:07.830777+0000 (oldest deadline 2024-09-21T12:36:31.231331+0000) 2024-09-21T12:36:57.177 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 12:36:56 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-7[42127]: 2024-09-21T12:36:56.806+0000 7f552c86a640 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2024-09-21T12:36:17.132004+0000 front 2024-09-21T12:36:17.131993+0000 (oldest deadline 2024-09-21T12:36:41.232131+0000) 2024-09-21T12:36:57.178 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 12:36:56 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-7[42127]: 2024-09-21T12:36:56.806+0000 7f552c86a640 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.79:6822 osd.2 since back 2024-09-21T12:36:21.232698+0000 front 2024-09-21T12:36:21.232643+0000 (oldest deadline 2024-09-21T12:36:45.932532+0000) 2024-09-21T12:36:57.178 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 12:36:56 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-7[42127]: 2024-09-21T12:36:56.806+0000 7f552c86a640 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.79:6830 osd.3 since back 2024-09-21T12:36:30.633479+0000 front 2024-09-21T12:36:30.633439+0000 (oldest deadline 2024-09-21T12:36:55.333430+0000) 2024-09-21T12:36:58.177 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 12:36:57 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-7[42127]: 2024-09-21T12:36:57.791+0000 7f552c86a640 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-09-21T12:36:07.830828+0000 front 2024-09-21T12:36:07.830777+0000 (oldest deadline 2024-09-21T12:36:31.231331+0000) 2024-09-21T12:36:58.177 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 12:36:57 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-7[42127]: 2024-09-21T12:36:57.791+0000 7f552c86a640 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2024-09-21T12:36:17.132004+0000 front 2024-09-21T12:36:17.131993+0000 (oldest deadline 2024-09-21T12:36:41.232131+0000) 2024-09-21T12:36:58.177 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 12:36:57 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-7[42127]: 2024-09-21T12:36:57.791+0000 7f552c86a640 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.79:6822 osd.2 since back 2024-09-21T12:36:21.232698+0000 front 2024-09-21T12:36:21.232643+0000 (oldest deadline 2024-09-21T12:36:45.932532+0000) 2024-09-21T12:36:58.177 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 12:36:57 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-7[42127]: 2024-09-21T12:36:57.791+0000 7f552c86a640 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.79:6830 osd.3 since back 2024-09-21T12:36:30.633479+0000 front 2024-09-21T12:36:30.633439+0000 (oldest deadline 2024-09-21T12:36:55.333430+0000) 2024-09-21T12:36:58.177 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 12:36:57 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-6[38758]: 2024-09-21T12:36:57.751+0000 7f19815f0640 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-09-21T12:36:08.568374+0000 front 2024-09-21T12:36:08.568570+0000 (oldest deadline 2024-09-21T12:36:33.268485+0000) 2024-09-21T12:36:58.178 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 12:36:57 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-6[38758]: 2024-09-21T12:36:57.751+0000 7f19815f0640 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2024-09-21T12:36:16.069952+0000 front 2024-09-21T12:36:16.069931+0000 (oldest deadline 2024-09-21T12:36:40.769786+0000) 2024-09-21T12:36:58.178 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 12:36:57 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-6[38758]: 2024-09-21T12:36:57.751+0000 7f19815f0640 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.79:6822 osd.2 since back 2024-09-21T12:36:20.770367+0000 front 2024-09-21T12:36:20.770384+0000 (oldest deadline 2024-09-21T12:36:45.470242+0000) 2024-09-21T12:36:58.178 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 12:36:57 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-6[38758]: 2024-09-21T12:36:57.752+0000 7f19815f0640 -1 osd.6 111 heartbeat_check: no reply from 172.21.15.79:6830 osd.3 since back 2024-09-21T12:36:32.372038+0000 front 2024-09-21T12:36:32.371955+0000 (oldest deadline 2024-09-21T12:36:57.071979+0000) 2024-09-21T12:36:58.178 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 12:36:57 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-6[38758]: 2024-09-21T12:36:57.752+0000 7f19815f0640 -1 osd.6 111 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.24526.0:928 4.1f 4:f95f44c2:::notify.0:head [watch ping cookie 94516886903552] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e111) 2024-09-21T12:36:58.921 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 12:36:58 smithi104 podman[67032]: 2024-09-21 12:36:58.65907084 +0000 UTC m=+5.567220810 container died fd2852c1818e6dc9a450317abd3c92a7d228e53fbe1d83a2fc05393967476cb7 (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-33168022-7814-11ef-baf6-efdc52797490-osd-6, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=reef-6a1bdf8, org.label-schema.build-date=20240731) 2024-09-21T12:36:58.921 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 12:36:58 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-7[42127]: 2024-09-21T12:36:58.824+0000 7f552c86a640 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-09-21T12:36:07.830828+0000 front 2024-09-21T12:36:07.830777+0000 (oldest deadline 2024-09-21T12:36:31.231331+0000) 2024-09-21T12:36:58.921 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 12:36:58 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-7[42127]: 2024-09-21T12:36:58.824+0000 7f552c86a640 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2024-09-21T12:36:17.132004+0000 front 2024-09-21T12:36:17.131993+0000 (oldest deadline 2024-09-21T12:36:41.232131+0000) 2024-09-21T12:36:58.921 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 12:36:58 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-7[42127]: 2024-09-21T12:36:58.824+0000 7f552c86a640 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.79:6822 osd.2 since back 2024-09-21T12:36:21.232698+0000 front 2024-09-21T12:36:21.232643+0000 (oldest deadline 2024-09-21T12:36:45.932532+0000) 2024-09-21T12:36:58.921 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 12:36:58 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-7[42127]: 2024-09-21T12:36:58.824+0000 7f552c86a640 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.79:6830 osd.3 since back 2024-09-21T12:36:30.633479+0000 front 2024-09-21T12:36:30.633439+0000 (oldest deadline 2024-09-21T12:36:55.333430+0000) 2024-09-21T12:36:59.177 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 12:36:59 smithi104 podman[67032]: 2024-09-21 12:36:59.018924192 +0000 UTC m=+5.927074169 container cleanup fd2852c1818e6dc9a450317abd3c92a7d228e53fbe1d83a2fc05393967476cb7 (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-33168022-7814-11ef-baf6-efdc52797490-osd-6, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=reef-6a1bdf8, ceph=True) 2024-09-21T12:36:59.177 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 12:36:59 smithi104 bash[67032]: ceph-33168022-7814-11ef-baf6-efdc52797490-osd-6 2024-09-21T12:36:59.588 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 12:36:59 smithi104 podman[67050]: 2024-09-21 12:36:59.270096366 +0000 UTC m=+0.603787736 container remove fd2852c1818e6dc9a450317abd3c92a7d228e53fbe1d83a2fc05393967476cb7 (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-33168022-7814-11ef-baf6-efdc52797490-osd-6, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, RELEASE=reef-6a1bdf8, io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T12:36:59.844 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 12:36:59 smithi104 podman[67135]: 2024-09-21 12:36:59.58731419 +0000 UTC m=+0.108740364 container create f6823ed9ce4b596e4f931035882bb7daba55190e25733e489ac34b6e7e96e7d3 (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-33168022-7814-11ef-baf6-efdc52797490-osd-6-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=reef-6a1bdf8, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T12:36:59.844 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 12:36:59 smithi104 podman[67135]: 2024-09-21 12:36:59.495346799 +0000 UTC m=+0.016772984 image pull 0adab276c97e6f3143ffbca5eeff37fcd7dc3b5c0f339a9f747381d9993602de quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6 2024-09-21T12:36:59.844 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 12:36:59 smithi104 podman[67135]: 2024-09-21 12:36:59.789706452 +0000 UTC m=+0.311132626 container init f6823ed9ce4b596e4f931035882bb7daba55190e25733e489ac34b6e7e96e7d3 (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-33168022-7814-11ef-baf6-efdc52797490-osd-6-deactivate, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=reef-6a1bdf8, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-09-21T12:36:59.844 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 12:36:59 smithi104 podman[67135]: 2024-09-21 12:36:59.79287634 +0000 UTC m=+0.314302509 container start f6823ed9ce4b596e4f931035882bb7daba55190e25733e489ac34b6e7e96e7d3 (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-33168022-7814-11ef-baf6-efdc52797490-osd-6-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, ceph=True, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , RELEASE=reef-6a1bdf8, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2) 2024-09-21T12:37:00.177 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 12:36:59 smithi104 podman[67135]: 2024-09-21 12:36:59.879247386 +0000 UTC m=+0.400673573 container attach f6823ed9ce4b596e4f931035882bb7daba55190e25733e489ac34b6e7e96e7d3 (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-33168022-7814-11ef-baf6-efdc52797490-osd-6-deactivate, RELEASE=reef-6a1bdf8, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T12:37:00.177 INFO:journalctl@ceph.osd.6.smithi104.stdout:Sep 21 12:36:59 smithi104 podman[67135]: 2024-09-21 12:36:59.988470097 +0000 UTC m=+0.509896273 container died f6823ed9ce4b596e4f931035882bb7daba55190e25733e489ac34b6e7e96e7d3 (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-33168022-7814-11ef-baf6-efdc52797490-osd-6-deactivate, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=reef-6a1bdf8, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T12:37:00.177 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 12:36:59 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-7[42127]: 2024-09-21T12:36:59.842+0000 7f552c86a640 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-09-21T12:36:07.830828+0000 front 2024-09-21T12:36:07.830777+0000 (oldest deadline 2024-09-21T12:36:31.231331+0000) 2024-09-21T12:37:00.178 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 12:36:59 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-7[42127]: 2024-09-21T12:36:59.842+0000 7f552c86a640 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2024-09-21T12:36:17.132004+0000 front 2024-09-21T12:36:17.131993+0000 (oldest deadline 2024-09-21T12:36:41.232131+0000) 2024-09-21T12:37:00.178 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 12:36:59 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-7[42127]: 2024-09-21T12:36:59.842+0000 7f552c86a640 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.79:6822 osd.2 since back 2024-09-21T12:36:21.232698+0000 front 2024-09-21T12:36:21.232643+0000 (oldest deadline 2024-09-21T12:36:45.932532+0000) 2024-09-21T12:37:00.178 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 12:36:59 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-7[42127]: 2024-09-21T12:36:59.842+0000 7f552c86a640 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.79:6830 osd.3 since back 2024-09-21T12:36:30.633479+0000 front 2024-09-21T12:36:30.633439+0000 (oldest deadline 2024-09-21T12:36:55.333430+0000) 2024-09-21T12:37:00.621 DEBUG:teuthology.orchestra.run.smithi104:> sudo pkill -f 'journalctl -f -n 0 -u ceph-33168022-7814-11ef-baf6-efdc52797490@osd.6.service' 2024-09-21T12:37:00.662 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T12:37:00.663 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-09-21T12:37:00.663 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-09-21T12:37:00.663 DEBUG:teuthology.orchestra.run.smithi104:> sudo systemctl stop ceph-33168022-7814-11ef-baf6-efdc52797490@osd.7 2024-09-21T12:37:00.861 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 12:37:00 smithi104 systemd[1]: Stopping Ceph osd.7 for 33168022-7814-11ef-baf6-efdc52797490... 2024-09-21T12:37:01.163 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 12:37:00 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-7[42127]: 2024-09-21T12:37:00.891+0000 7f552c86a640 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-09-21T12:36:07.830828+0000 front 2024-09-21T12:36:07.830777+0000 (oldest deadline 2024-09-21T12:36:31.231331+0000) 2024-09-21T12:37:01.163 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 12:37:00 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-7[42127]: 2024-09-21T12:37:00.891+0000 7f552c86a640 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2024-09-21T12:36:17.132004+0000 front 2024-09-21T12:36:17.131993+0000 (oldest deadline 2024-09-21T12:36:41.232131+0000) 2024-09-21T12:37:01.163 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 12:37:00 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-7[42127]: 2024-09-21T12:37:00.891+0000 7f552c86a640 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.79:6822 osd.2 since back 2024-09-21T12:36:21.232698+0000 front 2024-09-21T12:36:21.232643+0000 (oldest deadline 2024-09-21T12:36:45.932532+0000) 2024-09-21T12:37:01.163 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 12:37:00 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-7[42127]: 2024-09-21T12:37:00.891+0000 7f552c86a640 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.79:6830 osd.3 since back 2024-09-21T12:36:30.633479+0000 front 2024-09-21T12:36:30.633439+0000 (oldest deadline 2024-09-21T12:36:55.333430+0000) 2024-09-21T12:37:02.049 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 12:37:01 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-7[42127]: 2024-09-21T12:37:01.873+0000 7f552c86a640 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-09-21T12:36:07.830828+0000 front 2024-09-21T12:36:07.830777+0000 (oldest deadline 2024-09-21T12:36:31.231331+0000) 2024-09-21T12:37:02.049 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 12:37:01 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-7[42127]: 2024-09-21T12:37:01.874+0000 7f552c86a640 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2024-09-21T12:36:17.132004+0000 front 2024-09-21T12:36:17.131993+0000 (oldest deadline 2024-09-21T12:36:41.232131+0000) 2024-09-21T12:37:02.049 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 12:37:01 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-7[42127]: 2024-09-21T12:37:01.874+0000 7f552c86a640 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.79:6822 osd.2 since back 2024-09-21T12:36:21.232698+0000 front 2024-09-21T12:36:21.232643+0000 (oldest deadline 2024-09-21T12:36:45.932532+0000) 2024-09-21T12:37:02.049 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 12:37:01 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-7[42127]: 2024-09-21T12:37:01.874+0000 7f552c86a640 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.79:6830 osd.3 since back 2024-09-21T12:36:30.633479+0000 front 2024-09-21T12:36:30.633439+0000 (oldest deadline 2024-09-21T12:36:55.333430+0000) 2024-09-21T12:37:02.302 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 12:37:02 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-7[42127]: 2024-09-21T12:37:02.136+0000 7f5530256640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T12:37:02.302 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 12:37:02 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-7[42127]: 2024-09-21T12:37:02.136+0000 7f5530256640 -1 osd.7 111 *** Got signal Terminated *** 2024-09-21T12:37:02.302 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 12:37:02 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-7[42127]: 2024-09-21T12:37:02.136+0000 7f5530256640 -1 osd.7 111 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T12:37:03.176 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 12:37:02 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-7[42127]: 2024-09-21T12:37:02.888+0000 7f552c86a640 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-09-21T12:36:07.830828+0000 front 2024-09-21T12:36:07.830777+0000 (oldest deadline 2024-09-21T12:36:31.231331+0000) 2024-09-21T12:37:03.177 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 12:37:02 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-7[42127]: 2024-09-21T12:37:02.888+0000 7f552c86a640 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2024-09-21T12:36:17.132004+0000 front 2024-09-21T12:36:17.131993+0000 (oldest deadline 2024-09-21T12:36:41.232131+0000) 2024-09-21T12:37:03.177 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 12:37:02 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-7[42127]: 2024-09-21T12:37:02.888+0000 7f552c86a640 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.79:6822 osd.2 since back 2024-09-21T12:36:21.232698+0000 front 2024-09-21T12:36:21.232643+0000 (oldest deadline 2024-09-21T12:36:45.932532+0000) 2024-09-21T12:37:03.177 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 12:37:02 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-7[42127]: 2024-09-21T12:37:02.888+0000 7f552c86a640 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.79:6830 osd.3 since back 2024-09-21T12:36:30.633479+0000 front 2024-09-21T12:36:30.633439+0000 (oldest deadline 2024-09-21T12:36:55.333430+0000) 2024-09-21T12:37:04.177 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 12:37:03 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-7[42127]: 2024-09-21T12:37:03.861+0000 7f552c86a640 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-09-21T12:36:07.830828+0000 front 2024-09-21T12:36:07.830777+0000 (oldest deadline 2024-09-21T12:36:31.231331+0000) 2024-09-21T12:37:04.177 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 12:37:03 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-7[42127]: 2024-09-21T12:37:03.861+0000 7f552c86a640 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2024-09-21T12:36:17.132004+0000 front 2024-09-21T12:36:17.131993+0000 (oldest deadline 2024-09-21T12:36:41.232131+0000) 2024-09-21T12:37:04.177 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 12:37:03 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-7[42127]: 2024-09-21T12:37:03.861+0000 7f552c86a640 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.79:6822 osd.2 since back 2024-09-21T12:36:21.232698+0000 front 2024-09-21T12:36:21.232643+0000 (oldest deadline 2024-09-21T12:36:45.932532+0000) 2024-09-21T12:37:04.177 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 12:37:03 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-7[42127]: 2024-09-21T12:37:03.861+0000 7f552c86a640 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.79:6830 osd.3 since back 2024-09-21T12:36:30.633479+0000 front 2024-09-21T12:36:30.633439+0000 (oldest deadline 2024-09-21T12:36:55.333430+0000) 2024-09-21T12:37:05.176 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 12:37:04 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-7[42127]: 2024-09-21T12:37:04.826+0000 7f552c86a640 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-09-21T12:36:07.830828+0000 front 2024-09-21T12:36:07.830777+0000 (oldest deadline 2024-09-21T12:36:31.231331+0000) 2024-09-21T12:37:05.177 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 12:37:04 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-7[42127]: 2024-09-21T12:37:04.826+0000 7f552c86a640 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2024-09-21T12:36:17.132004+0000 front 2024-09-21T12:36:17.131993+0000 (oldest deadline 2024-09-21T12:36:41.232131+0000) 2024-09-21T12:37:05.177 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 12:37:04 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-7[42127]: 2024-09-21T12:37:04.826+0000 7f552c86a640 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.79:6822 osd.2 since back 2024-09-21T12:36:21.232698+0000 front 2024-09-21T12:36:21.232643+0000 (oldest deadline 2024-09-21T12:36:45.932532+0000) 2024-09-21T12:37:05.177 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 12:37:04 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-7[42127]: 2024-09-21T12:37:04.826+0000 7f552c86a640 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.79:6830 osd.3 since back 2024-09-21T12:36:30.633479+0000 front 2024-09-21T12:36:30.633439+0000 (oldest deadline 2024-09-21T12:36:55.333430+0000) 2024-09-21T12:37:06.176 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 12:37:05 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-7[42127]: 2024-09-21T12:37:05.824+0000 7f552c86a640 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-09-21T12:36:07.830828+0000 front 2024-09-21T12:36:07.830777+0000 (oldest deadline 2024-09-21T12:36:31.231331+0000) 2024-09-21T12:37:06.177 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 12:37:05 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-7[42127]: 2024-09-21T12:37:05.824+0000 7f552c86a640 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2024-09-21T12:36:17.132004+0000 front 2024-09-21T12:36:17.131993+0000 (oldest deadline 2024-09-21T12:36:41.232131+0000) 2024-09-21T12:37:06.177 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 12:37:05 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-7[42127]: 2024-09-21T12:37:05.824+0000 7f552c86a640 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.79:6822 osd.2 since back 2024-09-21T12:36:21.232698+0000 front 2024-09-21T12:36:21.232643+0000 (oldest deadline 2024-09-21T12:36:45.932532+0000) 2024-09-21T12:37:06.177 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 12:37:05 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-7[42127]: 2024-09-21T12:37:05.824+0000 7f552c86a640 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.79:6830 osd.3 since back 2024-09-21T12:36:30.633479+0000 front 2024-09-21T12:36:30.633439+0000 (oldest deadline 2024-09-21T12:36:55.333430+0000) 2024-09-21T12:37:07.177 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 12:37:06 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-7[42127]: 2024-09-21T12:37:06.831+0000 7f552c86a640 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.79:6806 osd.0 since back 2024-09-21T12:36:07.830828+0000 front 2024-09-21T12:36:07.830777+0000 (oldest deadline 2024-09-21T12:36:31.231331+0000) 2024-09-21T12:37:07.177 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 12:37:06 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-7[42127]: 2024-09-21T12:37:06.831+0000 7f552c86a640 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.79:6814 osd.1 since back 2024-09-21T12:36:17.132004+0000 front 2024-09-21T12:36:17.131993+0000 (oldest deadline 2024-09-21T12:36:41.232131+0000) 2024-09-21T12:37:07.177 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 12:37:06 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-7[42127]: 2024-09-21T12:37:06.831+0000 7f552c86a640 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.79:6822 osd.2 since back 2024-09-21T12:36:21.232698+0000 front 2024-09-21T12:36:21.232643+0000 (oldest deadline 2024-09-21T12:36:45.932532+0000) 2024-09-21T12:37:07.177 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 12:37:06 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-osd-7[42127]: 2024-09-21T12:37:06.831+0000 7f552c86a640 -1 osd.7 111 heartbeat_check: no reply from 172.21.15.79:6830 osd.3 since back 2024-09-21T12:36:30.633479+0000 front 2024-09-21T12:36:30.633439+0000 (oldest deadline 2024-09-21T12:36:55.333430+0000) 2024-09-21T12:37:07.959 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 12:37:07 smithi104 podman[67292]: 2024-09-21 12:37:07.689905889 +0000 UTC m=+5.833873195 container died d2b958c30fc4ea7fe4b91ccd0f582fce28e834e69d48848145779d0f7c1e5900 (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-33168022-7814-11ef-baf6-efdc52797490-osd-7, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, RELEASE=reef-6a1bdf8, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-09-21T12:37:08.327 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 12:37:08 smithi104 podman[67292]: 2024-09-21 12:37:08.140943356 +0000 UTC m=+6.284910701 container cleanup d2b958c30fc4ea7fe4b91ccd0f582fce28e834e69d48848145779d0f7c1e5900 (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-33168022-7814-11ef-baf6-efdc52797490-osd-7, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=reef-6a1bdf8, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-09-21T12:37:08.328 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 12:37:08 smithi104 bash[67292]: ceph-33168022-7814-11ef-baf6-efdc52797490-osd-7 2024-09-21T12:37:09.117 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 12:37:08 smithi104 podman[67753]: 2024-09-21 12:37:08.802412168 +0000 UTC m=+1.215002147 container remove d2b958c30fc4ea7fe4b91ccd0f582fce28e834e69d48848145779d0f7c1e5900 (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-33168022-7814-11ef-baf6-efdc52797490-osd-7, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=reef-6a1bdf8, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-09-21T12:37:09.117 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 12:37:08 smithi104 bash[67841]: Error: no container with ID d2b958c30fc4ea7fe4b91ccd0f582fce28e834e69d48848145779d0f7c1e5900 found in database: no such container 2024-09-21T12:37:09.842 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 12:37:09 smithi104 podman[68000]: 2024-09-21 12:37:09.449816579 +0000 UTC m=+0.019300314 image pull 0adab276c97e6f3143ffbca5eeff37fcd7dc3b5c0f339a9f747381d9993602de quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6 2024-09-21T12:37:09.843 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 12:37:09 smithi104 podman[68000]: 2024-09-21 12:37:09.547604993 +0000 UTC m=+0.117088752 container create a700a585f0173c1446f05dceee83c0d5537e875db98e4789dbe0860e38e784b7 (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-33168022-7814-11ef-baf6-efdc52797490-osd-7-deactivate, RELEASE=reef-6a1bdf8, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.37.2) 2024-09-21T12:37:11.427 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 12:37:11 smithi104 podman[68000]: 2024-09-21 12:37:11.005718347 +0000 UTC m=+1.575202069 container init a700a585f0173c1446f05dceee83c0d5537e875db98e4789dbe0860e38e784b7 (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-33168022-7814-11ef-baf6-efdc52797490-osd-7-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=reef-6a1bdf8, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-09-21T12:37:11.427 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 12:37:11 smithi104 podman[68000]: 2024-09-21 12:37:11.008967109 +0000 UTC m=+1.578450827 container start a700a585f0173c1446f05dceee83c0d5537e875db98e4789dbe0860e38e784b7 (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-33168022-7814-11ef-baf6-efdc52797490-osd-7-deactivate, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , RELEASE=reef-6a1bdf8, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True) 2024-09-21T12:37:11.427 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 12:37:11 smithi104 podman[68000]: 2024-09-21 12:37:11.088720609 +0000 UTC m=+1.658204344 container attach a700a585f0173c1446f05dceee83c0d5537e875db98e4789dbe0860e38e784b7 (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-33168022-7814-11ef-baf6-efdc52797490-osd-7-deactivate, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, RELEASE=reef-6a1bdf8, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731) 2024-09-21T12:37:11.427 INFO:journalctl@ceph.osd.7.smithi104.stdout:Sep 21 12:37:11 smithi104 podman[68000]: 2024-09-21 12:37:11.198629457 +0000 UTC m=+1.768113190 container died a700a585f0173c1446f05dceee83c0d5537e875db98e4789dbe0860e38e784b7 (image=quay.ceph.io/ceph-ci/ceph@sha256:de4fa798d334aaeb6a820ab2dad2fbcb86bcd7f118d94ed3a02ce126be31cec6, name=ceph-33168022-7814-11ef-baf6-efdc52797490-osd-7-deactivate, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, ceph=True, RELEASE=reef-6a1bdf8, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T12:37:11.847 DEBUG:teuthology.orchestra.run.smithi104:> sudo pkill -f 'journalctl -f -n 0 -u ceph-33168022-7814-11ef-baf6-efdc52797490@osd.7.service' 2024-09-21T12:37:11.888 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T12:37:11.889 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-09-21T12:37:11.889 INFO:tasks.cephadm.prometheus.a:Stopping prometheus.a... 2024-09-21T12:37:11.889 DEBUG:teuthology.orchestra.run.smithi104:> sudo systemctl stop ceph-33168022-7814-11ef-baf6-efdc52797490@prometheus.a 2024-09-21T12:37:12.176 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:37:11 smithi104 systemd[1]: Stopping Ceph prometheus.a for 33168022-7814-11ef-baf6-efdc52797490... 2024-09-21T12:37:12.488 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:37:12 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[63610]: ts=2024-09-21T12:37:12.228Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-09-21T12:37:12.488 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:37:12 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[63610]: ts=2024-09-21T12:37:12.228Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-09-21T12:37:12.488 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:37:12 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[63610]: ts=2024-09-21T12:37:12.228Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-09-21T12:37:12.488 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:37:12 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[63610]: ts=2024-09-21T12:37:12.228Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-09-21T12:37:12.488 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:37:12 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[63610]: ts=2024-09-21T12:37:12.228Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-09-21T12:37:12.489 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:37:12 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[63610]: ts=2024-09-21T12:37:12.228Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-09-21T12:37:12.489 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:37:12 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[63610]: ts=2024-09-21T12:37:12.228Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-09-21T12:37:12.489 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:37:12 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[63610]: ts=2024-09-21T12:37:12.228Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-09-21T12:37:12.489 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:37:12 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[63610]: ts=2024-09-21T12:37:12.229Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-09-21T12:37:12.489 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:37:12 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[63610]: ts=2024-09-21T12:37:12.229Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-09-21T12:37:12.489 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:37:12 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[63610]: ts=2024-09-21T12:37:12.229Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-09-21T12:37:12.489 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:37:12 smithi104 ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a[63610]: ts=2024-09-21T12:37:12.229Z caller=main.go:1147 level=info msg="See you next time!" 2024-09-21T12:37:12.489 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:37:12 smithi104 podman[68249]: 2024-09-21 12:37:12.239053018 +0000 UTC m=+0.125909621 container died 79f2e38dc5df4d3dc9939222b5389c29e22dd165c870e7e5dd1ed78d68acad76 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T12:37:12.921 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:37:12 smithi104 podman[68249]: 2024-09-21 12:37:12.594760902 +0000 UTC m=+0.481617502 container cleanup 79f2e38dc5df4d3dc9939222b5389c29e22dd165c870e7e5dd1ed78d68acad76 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T12:37:12.921 INFO:journalctl@ceph.prometheus.a.smithi104.stdout:Sep 21 12:37:12 smithi104 bash[68249]: ceph-33168022-7814-11ef-baf6-efdc52797490-prometheus-a 2024-09-21T12:37:13.068 DEBUG:teuthology.orchestra.run.smithi104:> sudo pkill -f 'journalctl -f -n 0 -u ceph-33168022-7814-11ef-baf6-efdc52797490@prometheus.a.service' 2024-09-21T12:37:13.103 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T12:37:13.103 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a 2024-09-21T12:37:13.104 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 33168022-7814-11ef-baf6-efdc52797490 --force --keep-logs 2024-09-21T12:37:13.259 INFO:teuthology.orchestra.run.smithi079.stdout:Deleting cluster with fsid: 33168022-7814-11ef-baf6-efdc52797490 2024-09-21T12:37:15.012 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Sep 21 12:37:14 smithi079 systemd[1]: Stopping Ceph alertmanager.a for 33168022-7814-11ef-baf6-efdc52797490... 2024-09-21T12:37:15.012 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Sep 21 12:37:14 smithi079 ceph-33168022-7814-11ef-baf6-efdc52797490-alertmanager-a[54523]: ts=2024-09-21T12:37:14.970Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-09-21T12:37:15.263 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Sep 21 12:37:15 smithi079 podman[84361]: 2024-09-21 12:37:15.011307754 +0000 UTC m=+0.131282305 container died 1451b4f42e50f68a2c8a80f740525dfa157c81c4af6d868c35121a760f6b3354 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-33168022-7814-11ef-baf6-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-21T12:37:15.597 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Sep 21 12:37:15 smithi079 podman[84361]: 2024-09-21 12:37:15.437597 +0000 UTC m=+0.557571506 container cleanup 1451b4f42e50f68a2c8a80f740525dfa157c81c4af6d868c35121a760f6b3354 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-33168022-7814-11ef-baf6-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-21T12:37:15.598 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Sep 21 12:37:15 smithi079 bash[84361]: ceph-33168022-7814-11ef-baf6-efdc52797490-alertmanager-a 2024-09-21T12:37:15.883 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Sep 21 12:37:15 smithi079 podman[84373]: 2024-09-21 12:37:15.597259877 +0000 UTC m=+0.577223141 container remove 1451b4f42e50f68a2c8a80f740525dfa157c81c4af6d868c35121a760f6b3354 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-33168022-7814-11ef-baf6-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-21T12:37:15.883 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Sep 21 12:37:15 smithi079 podman[84373]: 2024-09-21 12:37:15.68950982 +0000 UTC m=+0.669473083 volume remove fb7afddfd57c7dc4cf35ece272dff3a08e91c7994cbce1073329340ff0f6d750 2024-09-21T12:37:16.192 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Sep 21 12:37:16 smithi079 systemd[1]: Stopping Ceph node-exporter.a for 33168022-7814-11ef-baf6-efdc52797490... 2024-09-21T12:37:16.192 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Sep 21 12:37:15 smithi079 systemd[1]: ceph-33168022-7814-11ef-baf6-efdc52797490@alertmanager.a.service: Deactivated successfully. 2024-09-21T12:37:16.192 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Sep 21 12:37:15 smithi079 systemd[1]: Stopped Ceph alertmanager.a for 33168022-7814-11ef-baf6-efdc52797490. 2024-09-21T12:37:16.193 INFO:journalctl@ceph.alertmanager.a.smithi079.stdout:Sep 21 12:37:15 smithi079 systemd[1]: ceph-33168022-7814-11ef-baf6-efdc52797490@alertmanager.a.service: Consumed 1.362s CPU time. 2024-09-21T12:37:16.619 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Sep 21 12:37:16 smithi079 podman[84483]: 2024-09-21 12:37:16.344971907 +0000 UTC m=+0.114635169 container died f7bef67502dce5dc1f830f29a2e1911744e2a54a91efc7b6a8bb02a9e03c7f9e (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-33168022-7814-11ef-baf6-efdc52797490-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-21T12:37:16.941 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Sep 21 12:37:16 smithi079 podman[84483]: 2024-09-21 12:37:16.801278354 +0000 UTC m=+0.570941608 container cleanup f7bef67502dce5dc1f830f29a2e1911744e2a54a91efc7b6a8bb02a9e03c7f9e (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-33168022-7814-11ef-baf6-efdc52797490-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-21T12:37:16.942 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Sep 21 12:37:16 smithi079 bash[84483]: ceph-33168022-7814-11ef-baf6-efdc52797490-node-exporter-a 2024-09-21T12:37:17.387 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Sep 21 12:37:17 smithi079 podman[84497]: 2024-09-21 12:37:17.052954532 +0000 UTC m=+0.712040672 container remove f7bef67502dce5dc1f830f29a2e1911744e2a54a91efc7b6a8bb02a9e03c7f9e (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-33168022-7814-11ef-baf6-efdc52797490-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-21T12:37:17.387 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Sep 21 12:37:17 smithi079 systemd[1]: ceph-33168022-7814-11ef-baf6-efdc52797490@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-09-21T12:37:17.387 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Sep 21 12:37:17 smithi079 systemd[1]: ceph-33168022-7814-11ef-baf6-efdc52797490@node-exporter.a.service: Failed with result 'exit-code'. 2024-09-21T12:37:17.387 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Sep 21 12:37:17 smithi079 systemd[1]: Stopped Ceph node-exporter.a for 33168022-7814-11ef-baf6-efdc52797490. 2024-09-21T12:37:17.387 INFO:journalctl@ceph.node-exporter.a.smithi079.stdout:Sep 21 12:37:17 smithi079 systemd[1]: ceph-33168022-7814-11ef-baf6-efdc52797490@node-exporter.a.service: Consumed 2.435s CPU time. 2024-09-21T12:37:52.259 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 33168022-7814-11ef-baf6-efdc52797490 --force --keep-logs 2024-09-21T12:37:52.415 INFO:teuthology.orchestra.run.smithi104.stdout:Deleting cluster with fsid: 33168022-7814-11ef-baf6-efdc52797490 2024-09-21T12:37:54.007 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 12:37:53 smithi104 systemd[1]: Stopping Ceph node-exporter.b for 33168022-7814-11ef-baf6-efdc52797490... 2024-09-21T12:37:54.426 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 12:37:54 smithi104 podman[69660]: 2024-09-21 12:37:54.178892398 +0000 UTC m=+0.107505999 container died 3701f535d07838f1eab9dfdb3b98ec612ce03736f752b85af28317d7e811aac5 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-33168022-7814-11ef-baf6-efdc52797490-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-21T12:37:54.907 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 12:37:54 smithi104 podman[69660]: 2024-09-21 12:37:54.559152129 +0000 UTC m=+0.487765721 container cleanup 3701f535d07838f1eab9dfdb3b98ec612ce03736f752b85af28317d7e811aac5 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-33168022-7814-11ef-baf6-efdc52797490-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-21T12:37:54.908 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 12:37:54 smithi104 bash[69660]: ceph-33168022-7814-11ef-baf6-efdc52797490-node-exporter-b 2024-09-21T12:37:55.176 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 12:37:54 smithi104 podman[69672]: 2024-09-21 12:37:54.906830634 +0000 UTC m=+0.727032348 container remove 3701f535d07838f1eab9dfdb3b98ec612ce03736f752b85af28317d7e811aac5 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-33168022-7814-11ef-baf6-efdc52797490-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-21T12:37:55.177 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 12:37:54 smithi104 systemd[1]: ceph-33168022-7814-11ef-baf6-efdc52797490@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-09-21T12:37:55.177 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 12:37:55 smithi104 systemd[1]: ceph-33168022-7814-11ef-baf6-efdc52797490@node-exporter.b.service: Failed with result 'exit-code'. 2024-09-21T12:37:55.177 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 12:37:55 smithi104 systemd[1]: Stopped Ceph node-exporter.b for 33168022-7814-11ef-baf6-efdc52797490. 2024-09-21T12:37:55.177 INFO:journalctl@ceph.node-exporter.b.smithi104.stdout:Sep 21 12:37:55 smithi104 systemd[1]: ceph-33168022-7814-11ef-baf6-efdc52797490@node-exporter.b.service: Consumed 2.394s CPU time. 2024-09-21T12:37:55.646 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:37:55 smithi104 systemd[1]: Stopping Ceph grafana.a for 33168022-7814-11ef-baf6-efdc52797490... 2024-09-21T12:37:55.647 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:37:55 smithi104 podman[69780]: 2024-09-21 12:37:55.465825327 +0000 UTC m=+0.098767588 container died c85dd02f65193c80f3b7974b4ec16525c6f44314101dce6eee0eb10464c1d55c (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a, name=ubi8, io.openshift.tags=base rhel8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, build-date=2022-04-29T11:48:11.055897, com.redhat.component=ubi8-container, io.openshift.expose-services=, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=239.1651231664, architecture=x86_64, maintainer=Paul Cuzner , summary=Grafana Container configured for Ceph mgr/dashboard integration, vcs-type=git, description=Ceph Grafana Container, distribution-scope=public, version=8.5, io.buildah.version=1.32.0, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, vendor=Red Hat, Inc., io.k8s.display-name=Red Hat Universal Base Image 8) 2024-09-21T12:37:55.647 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:37:55 smithi104 podman[69780]: 2024-09-21 12:37:55.645767299 +0000 UTC m=+0.278709561 container exec_died c85dd02f65193c80f3b7974b4ec16525c6f44314101dce6eee0eb10464c1d55c (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, description=Ceph Grafana Container, architecture=x86_64, maintainer=Paul Cuzner , io.buildah.version=1.32.0, io.k8s.display-name=Red Hat Universal Base Image 8, io.openshift.expose-services=, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, release=239.1651231664, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, vendor=Red Hat, Inc., version=8.5, name=ubi8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, distribution-scope=public, vcs-type=git, com.redhat.component=ubi8-container, build-date=2022-04-29T11:48:11.055897, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.openshift.tags=base rhel8) 2024-09-21T12:37:55.926 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:37:55 smithi104 podman[69780]: 2024-09-21 12:37:55.855205266 +0000 UTC m=+0.488147524 container cleanup c85dd02f65193c80f3b7974b4ec16525c6f44314101dce6eee0eb10464c1d55c (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a, release=239.1651231664, maintainer=Paul Cuzner , architecture=x86_64, io.k8s.display-name=Red Hat Universal Base Image 8, description=Ceph Grafana Container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-04-29T11:48:11.055897, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.buildah.version=1.32.0, io.openshift.tags=base rhel8, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, vcs-type=git, com.redhat.component=ubi8-container, name=ubi8, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.openshift.expose-services=, vendor=Red Hat, Inc., version=8.5) 2024-09-21T12:37:55.927 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:37:55 smithi104 bash[69780]: ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a 2024-09-21T12:37:56.506 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:37:56 smithi104 podman[69792]: 2024-09-21 12:37:56.181714721 +0000 UTC m=+0.714870976 container remove c85dd02f65193c80f3b7974b4ec16525c6f44314101dce6eee0eb10464c1d55c (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-33168022-7814-11ef-baf6-efdc52797490-grafana-a, release=239.1651231664, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=Red Hat Universal Base Image 8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.component=ubi8-container, name=ubi8, build-date=2022-04-29T11:48:11.055897, description=Ceph Grafana Container, io.openshift.tags=base rhel8, version=8.5, distribution-scope=public, io.openshift.expose-services=, maintainer=Paul Cuzner , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, vcs-type=git, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.buildah.version=1.32.0, vendor=Red Hat, Inc., architecture=x86_64) 2024-09-21T12:37:56.506 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:37:56 smithi104 systemd[1]: ceph-33168022-7814-11ef-baf6-efdc52797490@grafana.a.service: Main process exited, code=exited, status=143/n/a 2024-09-21T12:37:56.506 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:37:56 smithi104 systemd[1]: ceph-33168022-7814-11ef-baf6-efdc52797490@grafana.a.service: Failed with result 'exit-code'. 2024-09-21T12:37:56.506 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:37:56 smithi104 systemd[1]: Stopped Ceph grafana.a for 33168022-7814-11ef-baf6-efdc52797490. 2024-09-21T12:37:56.506 INFO:journalctl@ceph.grafana.a.smithi104.stdout:Sep 21 12:37:56 smithi104 systemd[1]: ceph-33168022-7814-11ef-baf6-efdc52797490@grafana.a.service: Consumed 8.652s CPU time. 2024-09-21T12:38:19.244 DEBUG:teuthology.orchestra.run.smithi079:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-21T12:38:19.279 DEBUG:teuthology.orchestra.run.smithi104:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-21T12:38:19.307 INFO:tasks.cephadm:Archiving crash dumps... 2024-09-21T12:38:19.308 DEBUG:teuthology.misc:Transferring archived files from smithi079:/var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/crash to /home/teuthworker/archive/gabrioux-2024-09-21_11:23:40-orch:cephadm-wip-guits-main-2024-09-21-0842-distro-default-smithi/7913934/remote/smithi079/crash 2024-09-21T12:38:19.309 DEBUG:teuthology.orchestra.run.smithi079:> sudo tar c -f - -C /var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/crash -- . 2024-09-21T12:38:19.396 INFO:teuthology.orchestra.run.smithi079.stderr:tar: /var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/crash: Cannot open: No such file or directory 2024-09-21T12:38:19.396 INFO:teuthology.orchestra.run.smithi079.stderr:tar: Error is not recoverable: exiting now 2024-09-21T12:38:19.398 DEBUG:teuthology.misc:Transferring archived files from smithi104:/var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/crash to /home/teuthworker/archive/gabrioux-2024-09-21_11:23:40-orch:cephadm-wip-guits-main-2024-09-21-0842-distro-default-smithi/7913934/remote/smithi104/crash 2024-09-21T12:38:19.399 DEBUG:teuthology.orchestra.run.smithi104:> sudo tar c -f - -C /var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/crash -- . 2024-09-21T12:38:19.430 INFO:teuthology.orchestra.run.smithi104.stderr:tar: /var/lib/ceph/33168022-7814-11ef-baf6-efdc52797490/crash: Cannot open: No such file or directory 2024-09-21T12:38:19.430 INFO:teuthology.orchestra.run.smithi104.stderr:tar: Error is not recoverable: exiting now 2024-09-21T12:38:19.431 INFO:tasks.cephadm:Checking cluster log for badness... 2024-09-21T12:38:19.432 DEBUG:teuthology.orchestra.run.smithi079:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/33168022-7814-11ef-baf6-efdc52797490/ceph.log | egrep CEPHADM_ | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v CEPHADM_STRAY_DAEMON | egrep -v CEPHADM_FAILED_DAEMON | egrep -v CEPHADM_AGENT_DOWN | head -n 1 2024-09-21T12:38:19.464 INFO:tasks.cephadm:Compressing logs... 2024-09-21T12:38:19.464 DEBUG:teuthology.orchestra.run.smithi079:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-21T12:38:19.507 DEBUG:teuthology.orchestra.run.smithi104:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-21T12:38:19.530 INFO:teuthology.orchestra.run.smithi079.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/33168022-7814-11ef-baf6-efdc52797490/ceph-osd.0.log 2024-09-21T12:38:19.531 INFO:teuthology.orchestra.run.smithi079.stderr:: No such file or directory 2024-09-21T12:38:19.539 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/33168022-7814-11ef-baf6-efdc52797490/tcmu-runner.log 2024-09-21T12:38:19.539 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/33168022-7814-11ef-baf6-efdc52797490/ceph-osd.1.log 2024-09-21T12:38:19.539 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/33168022-7814-11ef-baf6-efdc52797490/ceph-osd.0.log: /var/log/ceph/33168022-7814-11ef-baf6-efdc52797490/tcmu-runner.log: gzip -5 --verbose -- /var/log/ceph/33168022-7814-11ef-baf6-efdc52797490/ceph-volume.log 2024-09-21T12:38:19.539 INFO:teuthology.orchestra.run.smithi079.stderr: 86.5% -- replaced with /var/log/ceph/33168022-7814-11ef-baf6-efdc52797490/tcmu-runner.log.gz 2024-09-21T12:38:19.539 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/33168022-7814-11ef-baf6-efdc52797490/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/33168022-7814-11ef-baf6-efdc52797490/ceph.audit.log 2024-09-21T12:38:19.539 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/33168022-7814-11ef-baf6-efdc52797490/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/33168022-7814-11ef-baf6-efdc52797490/ceph-mon.c.log 2024-09-21T12:38:19.540 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/33168022-7814-11ef-baf6-efdc52797490/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/33168022-7814-11ef-baf6-efdc52797490/ceph.cephadm.log 2024-09-21T12:38:19.540 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/33168022-7814-11ef-baf6-efdc52797490/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/33168022-7814-11ef-baf6-efdc52797490/ceph-client.rgw.smpl.smithi079.tydkfh.log 2024-09-21T12:38:19.540 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/33168022-7814-11ef-baf6-efdc52797490/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/33168022-7814-11ef-baf6-efdc52797490/ceph-mon.a.log 2024-09-21T12:38:19.540 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/33168022-7814-11ef-baf6-efdc52797490/ceph-client.rgw.smpl.smithi079.tydkfh.log: 62.2% -- replaced with /var/log/ceph/33168022-7814-11ef-baf6-efdc52797490/ceph-client.rgw.smpl.smithi079.tydkfh.log.gz 2024-09-21T12:38:19.540 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/33168022-7814-11ef-baf6-efdc52797490/ceph-mon.a.log: 93.2% -- replaced with /var/log/ceph/33168022-7814-11ef-baf6-efdc52797490/ceph.cephadm.log.gz 2024-09-21T12:38:19.540 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/33168022-7814-11ef-baf6-efdc52797490/ceph-osd.2.log 2024-09-21T12:38:19.540 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/33168022-7814-11ef-baf6-efdc52797490/ceph-client.rgw.foo.smithi079.dfrddk.log 2024-09-21T12:38:19.541 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/33168022-7814-11ef-baf6-efdc52797490/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/33168022-7814-11ef-baf6-efdc52797490/ceph-mgr.y.log 2024-09-21T12:38:19.541 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/33168022-7814-11ef-baf6-efdc52797490/ceph-client.rgw.foo.smithi079.dfrddk.log: 62.2% -- replaced with /var/log/ceph/33168022-7814-11ef-baf6-efdc52797490/ceph-client.rgw.foo.smithi079.dfrddk.log.gz 2024-09-21T12:38:19.541 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/33168022-7814-11ef-baf6-efdc52797490/ceph.log 2024-09-21T12:38:19.541 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/33168022-7814-11ef-baf6-efdc52797490/ceph-mgr.y.log: 94.2% -- replaced with /var/log/ceph/33168022-7814-11ef-baf6-efdc52797490/ceph.audit.log.gz 2024-09-21T12:38:19.542 INFO:teuthology.orchestra.run.smithi104.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/33168022-7814-11ef-baf6-efdc52797490/ceph-osd.7.log 2024-09-21T12:38:19.542 INFO:teuthology.orchestra.run.smithi104.stderr:: No such file or directory 2024-09-21T12:38:19.542 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/33168022-7814-11ef-baf6-efdc52797490/ceph-volume.log 2024-09-21T12:38:19.542 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/33168022-7814-11ef-baf6-efdc52797490/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/33168022-7814-11ef-baf6-efdc52797490/ceph-osd.4.log 2024-09-21T12:38:19.542 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/33168022-7814-11ef-baf6-efdc52797490/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/33168022-7814-11ef-baf6-efdc52797490/ceph.audit.log 2024-09-21T12:38:19.543 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/33168022-7814-11ef-baf6-efdc52797490/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/33168022-7814-11ef-baf6-efdc52797490/ceph.cephadm.log 2024-09-21T12:38:19.543 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/33168022-7814-11ef-baf6-efdc52797490/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/33168022-7814-11ef-baf6-efdc52797490/ceph-osd.6.log 2024-09-21T12:38:19.543 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/33168022-7814-11ef-baf6-efdc52797490/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/33168022-7814-11ef-baf6-efdc52797490/ceph-osd.5.log 2024-09-21T12:38:19.543 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/33168022-7814-11ef-baf6-efdc52797490/ceph-osd.6.log: 88.4% -- replaced with /var/log/ceph/33168022-7814-11ef-baf6-efdc52797490/ceph.cephadm.log.gz 2024-09-21T12:38:19.543 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/33168022-7814-11ef-baf6-efdc52797490/ceph.log 2024-09-21T12:38:19.543 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/33168022-7814-11ef-baf6-efdc52797490/ceph-osd.5.log: 90.7% -- replaced with /var/log/ceph/33168022-7814-11ef-baf6-efdc52797490/ceph.audit.log.gz 2024-09-21T12:38:19.543 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/33168022-7814-11ef-baf6-efdc52797490/ceph-mgr.x.log 2024-09-21T12:38:19.543 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/33168022-7814-11ef-baf6-efdc52797490/ceph.log: 87.5% -- replaced with /var/log/ceph/33168022-7814-11ef-baf6-efdc52797490/ceph.log.gz 2024-09-21T12:38:19.544 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/33168022-7814-11ef-baf6-efdc52797490/ceph-client.rgw.smpl.smithi104.riklnj.log 2024-09-21T12:38:19.544 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/33168022-7814-11ef-baf6-efdc52797490/ceph-osd.3.log 2024-09-21T12:38:19.551 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/33168022-7814-11ef-baf6-efdc52797490/ceph.log: 92.7% -- replaced with /var/log/ceph/33168022-7814-11ef-baf6-efdc52797490/ceph.log.gz 2024-09-21T12:38:19.552 INFO:teuthology.orchestra.run.smithi079.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-09-21T12:38:19.553 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/33168022-7814-11ef-baf6-efdc52797490/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/33168022-7814-11ef-baf6-efdc52797490/ceph-client.rgw.foo.smithi104.qonbxe.log 2024-09-21T12:38:19.553 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/33168022-7814-11ef-baf6-efdc52797490/ceph-client.rgw.smpl.smithi104.riklnj.log: 62.2% -- replaced with /var/log/ceph/33168022-7814-11ef-baf6-efdc52797490/ceph-client.rgw.smpl.smithi104.riklnj.log.gz 2024-09-21T12:38:19.561 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/33168022-7814-11ef-baf6-efdc52797490/ceph-mon.b.log 2024-09-21T12:38:19.564 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/33168022-7814-11ef-baf6-efdc52797490/ceph-client.rgw.foo.smithi104.qonbxe.log: 62.2% -- replaced with /var/log/ceph/33168022-7814-11ef-baf6-efdc52797490/ceph-client.rgw.foo.smithi104.qonbxe.log.gz 2024-09-21T12:38:19.573 INFO:teuthology.orchestra.run.smithi104.stderr: 91.4% -- replaced with /var/log/ceph/33168022-7814-11ef-baf6-efdc52797490/ceph-volume.log.gz 2024-09-21T12:38:19.574 INFO:teuthology.orchestra.run.smithi104.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-09-21T12:38:19.579 INFO:teuthology.orchestra.run.smithi079.stderr:/var/log/ceph/33168022-7814-11ef-baf6-efdc52797490/ceph-osd.3.log: /var/log/ceph/cephadm.log: 84.5% -- replaced with /var/log/ceph/cephadm.log.gz 2024-09-21T12:38:19.581 INFO:teuthology.orchestra.run.smithi079.stderr: 91.4% -- replaced with /var/log/ceph/33168022-7814-11ef-baf6-efdc52797490/ceph-volume.log.gz 2024-09-21T12:38:19.599 INFO:teuthology.orchestra.run.smithi104.stderr:/var/log/ceph/33168022-7814-11ef-baf6-efdc52797490/ceph-mon.b.log: /var/log/ceph/cephadm.log: 79.2% -- replaced with /var/log/ceph/cephadm.log.gz 2024-09-21T12:38:19.603 INFO:teuthology.orchestra.run.smithi104.stderr: 87.7% -- replaced with /var/log/ceph/33168022-7814-11ef-baf6-efdc52797490/ceph-mgr.x.log.gz 2024-09-21T12:38:19.729 INFO:teuthology.orchestra.run.smithi079.stderr: 92.3% -- replaced with /var/log/ceph/33168022-7814-11ef-baf6-efdc52797490/ceph-mon.c.log.gz 2024-09-21T12:38:19.777 INFO:teuthology.orchestra.run.smithi079.stderr: 87.7% -- replaced with /var/log/ceph/33168022-7814-11ef-baf6-efdc52797490/ceph-mgr.y.log.gz 2024-09-21T12:38:19.813 INFO:teuthology.orchestra.run.smithi104.stderr: 92.2% -- replaced with /var/log/ceph/33168022-7814-11ef-baf6-efdc52797490/ceph-mon.b.log.gz 2024-09-21T12:38:19.998 INFO:teuthology.orchestra.run.smithi079.stderr: 93.4% -- replaced with /var/log/ceph/33168022-7814-11ef-baf6-efdc52797490/ceph-osd.2.log.gz 2024-09-21T12:38:20.028 INFO:teuthology.orchestra.run.smithi104.stderr: 93.3% -- replaced with /var/log/ceph/33168022-7814-11ef-baf6-efdc52797490/ceph-osd.6.log.gz 2024-09-21T12:38:20.122 INFO:teuthology.orchestra.run.smithi104.stderr: 93.9% -- replaced with /var/log/ceph/33168022-7814-11ef-baf6-efdc52797490/ceph-osd.7.log.gz 2024-09-21T12:38:20.141 INFO:teuthology.orchestra.run.smithi079.stderr: 91.2% -- replaced with /var/log/ceph/33168022-7814-11ef-baf6-efdc52797490/ceph-mon.a.log.gz 2024-09-21T12:38:20.151 INFO:teuthology.orchestra.run.smithi104.stderr: 93.4% -- replaced with /var/log/ceph/33168022-7814-11ef-baf6-efdc52797490/ceph-osd.5.log.gz 2024-09-21T12:38:20.164 INFO:teuthology.orchestra.run.smithi079.stderr: 93.5% -- replaced with /var/log/ceph/33168022-7814-11ef-baf6-efdc52797490/ceph-osd.0.log.gz 2024-09-21T12:38:20.172 INFO:teuthology.orchestra.run.smithi079.stderr: 93.6% -- replaced with /var/log/ceph/33168022-7814-11ef-baf6-efdc52797490/ceph-osd.1.log.gz 2024-09-21T12:38:20.202 INFO:teuthology.orchestra.run.smithi104.stderr: 93.7% -- replaced with /var/log/ceph/33168022-7814-11ef-baf6-efdc52797490/ceph-osd.4.log.gz 2024-09-21T12:38:20.204 INFO:teuthology.orchestra.run.smithi104.stderr: 2024-09-21T12:38:20.204 INFO:teuthology.orchestra.run.smithi104.stderr:real 0m0.683s 2024-09-21T12:38:20.204 INFO:teuthology.orchestra.run.smithi104.stderr:user 0m2.623s 2024-09-21T12:38:20.204 INFO:teuthology.orchestra.run.smithi104.stderr:sys 0m0.120s 2024-09-21T12:38:20.209 INFO:teuthology.orchestra.run.smithi079.stderr: 93.6% -- replaced with /var/log/ceph/33168022-7814-11ef-baf6-efdc52797490/ceph-osd.3.log.gz 2024-09-21T12:38:20.211 INFO:teuthology.orchestra.run.smithi079.stderr: 2024-09-21T12:38:20.211 INFO:teuthology.orchestra.run.smithi079.stderr:real 0m0.691s 2024-09-21T12:38:20.211 INFO:teuthology.orchestra.run.smithi079.stderr:user 0m3.363s 2024-09-21T12:38:20.211 INFO:teuthology.orchestra.run.smithi079.stderr:sys 0m0.162s 2024-09-21T12:38:20.211 INFO:tasks.cephadm:Archiving logs... 2024-09-21T12:38:20.212 DEBUG:teuthology.misc:Transferring archived files from smithi079:/var/log/ceph to /home/teuthworker/archive/gabrioux-2024-09-21_11:23:40-orch:cephadm-wip-guits-main-2024-09-21-0842-distro-default-smithi/7913934/remote/smithi079/log 2024-09-21T12:38:20.213 DEBUG:teuthology.orchestra.run.smithi079:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-21T12:38:20.694 DEBUG:teuthology.misc:Transferring archived files from smithi104:/var/log/ceph to /home/teuthworker/archive/gabrioux-2024-09-21_11:23:40-orch:cephadm-wip-guits-main-2024-09-21-0842-distro-default-smithi/7913934/remote/smithi104/log 2024-09-21T12:38:20.695 DEBUG:teuthology.orchestra.run.smithi104:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-21T12:38:21.052 INFO:tasks.cephadm:Removing cluster... 2024-09-21T12:38:21.053 DEBUG:teuthology.orchestra.run.smithi079:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 33168022-7814-11ef-baf6-efdc52797490 --force 2024-09-21T12:38:21.196 INFO:teuthology.orchestra.run.smithi079.stdout:Deleting cluster with fsid: 33168022-7814-11ef-baf6-efdc52797490 2024-09-21T12:38:21.396 DEBUG:teuthology.orchestra.run.smithi104:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 33168022-7814-11ef-baf6-efdc52797490 --force 2024-09-21T12:38:21.544 INFO:teuthology.orchestra.run.smithi104.stdout:Deleting cluster with fsid: 33168022-7814-11ef-baf6-efdc52797490 2024-09-21T12:38:21.674 INFO:tasks.cephadm:Removing cephadm ... 2024-09-21T12:38:21.674 DEBUG:teuthology.orchestra.run.smithi079:> rm -rf /home/ubuntu/cephtest/cephadm 2024-09-21T12:38:21.689 DEBUG:teuthology.orchestra.run.smithi104:> rm -rf /home/ubuntu/cephtest/cephadm 2024-09-21T12:38:21.704 INFO:tasks.cephadm:Teardown complete 2024-09-21T12:38:21.704 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-21T12:38:21.714 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-21T12:38:21.714 DEBUG:teuthology.orchestra.run.smithi079:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-21T12:38:21.732 DEBUG:teuthology.orchestra.run.smithi104:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-21T12:38:21.743 INFO:teuthology.orchestra.run.smithi079.stderr:bash: line 1: ntpq: command not found 2024-09-21T12:38:21.747 INFO:teuthology.orchestra.run.smithi079.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-21T12:38:21.747 INFO:teuthology.orchestra.run.smithi079.stdout:=============================================================================== 2024-09-21T12:38:21.747 INFO:teuthology.orchestra.run.smithi079.stdout:^* hv01.front.sepia.ceph.com 3 6 377 27 +43us[ +52us] +/- 48ms 2024-09-21T12:38:21.748 INFO:teuthology.orchestra.run.smithi079.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 26 -261us[ -261us] +/- 62ms 2024-09-21T12:38:21.748 INFO:teuthology.orchestra.run.smithi079.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 27 +214us[ +214us] +/- 64ms 2024-09-21T12:38:21.748 INFO:teuthology.orchestra.run.smithi079.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T12:38:21.758 INFO:teuthology.orchestra.run.smithi104.stderr:bash: line 1: ntpq: command not found 2024-09-21T12:38:21.762 INFO:teuthology.orchestra.run.smithi104.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-21T12:38:21.762 INFO:teuthology.orchestra.run.smithi104.stdout:=============================================================================== 2024-09-21T12:38:21.762 INFO:teuthology.orchestra.run.smithi104.stdout:^* hv01.front.sepia.ceph.com 3 6 377 28 +36us[ +27us] +/- 48ms 2024-09-21T12:38:21.763 INFO:teuthology.orchestra.run.smithi104.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 28 -268us[ -268us] +/- 62ms 2024-09-21T12:38:21.763 INFO:teuthology.orchestra.run.smithi104.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 30 +221us[ +212us] +/- 64ms 2024-09-21T12:38:21.763 INFO:teuthology.orchestra.run.smithi104.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T12:38:21.764 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-21T12:38:21.772 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-21T12:38:21.772 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-21T12:38:21.779 DEBUG:teuthology.orchestra.run.smithi079:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-21T12:38:21.863 DEBUG:teuthology.orchestra.run.smithi104:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-21T12:38:21.932 DEBUG:teuthology.orchestra.run.smithi079:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-21T12:38:21.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:21.960 DEBUG:teuthology.orchestra.run.smithi104:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-21T12:38:21.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:21.988 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-21T12:38:21.996 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-21T12:38:22.004 INFO:teuthology.task.internal:Duration was 1161.527647 seconds 2024-09-21T12:38:22.004 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-21T12:38:22.012 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-21T12:38:22.012 DEBUG:teuthology.orchestra.run.smithi079:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-21T12:38:22.014 DEBUG:teuthology.orchestra.run.smithi104:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-21T12:38:22.050 INFO:teuthology.orchestra.run.smithi079.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-21T12:38:22.066 INFO:teuthology.orchestra.run.smithi104.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-21T12:38:22.464 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-21T12:38:22.464 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi079.front.sepia.ceph.com 2024-09-21T12:38:22.464 DEBUG:teuthology.orchestra.run.smithi079:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-21T12:38:22.485 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi104.front.sepia.ceph.com 2024-09-21T12:38:22.485 DEBUG:teuthology.orchestra.run.smithi104:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-21T12:38:22.532 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-21T12:38:22.532 DEBUG:teuthology.orchestra.run.smithi079:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-21T12:38:22.534 DEBUG:teuthology.orchestra.run.smithi104:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-21T12:38:22.598 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-21T12:38:22.598 DEBUG:teuthology.orchestra.run.smithi079:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-21T12:38:22.601 DEBUG:teuthology.orchestra.run.smithi104:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-21T12:38:22.914 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-21T12:38:22.923 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-21T12:38:22.923 DEBUG:teuthology.orchestra.run.smithi079:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-21T12:38:22.948 DEBUG:teuthology.orchestra.run.smithi104:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-21T12:38:22.972 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-21T12:38:22.981 DEBUG:teuthology.orchestra.run.smithi079:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-21T12:38:22.991 DEBUG:teuthology.orchestra.run.smithi104:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-21T12:38:23.011 INFO:teuthology.orchestra.run.smithi079.stdout:kernel.core_pattern = core 2024-09-21T12:38:23.035 INFO:teuthology.orchestra.run.smithi104.stdout:kernel.core_pattern = core 2024-09-21T12:38:23.060 DEBUG:teuthology.orchestra.run.smithi079:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-21T12:38:23.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:23.082 DEBUG:teuthology.orchestra.run.smithi104:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-21T12:38:23.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T12:38:23.114 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-21T12:38:23.123 INFO:teuthology.task.internal:Transferring archived files... 2024-09-21T12:38:23.124 DEBUG:teuthology.misc:Transferring archived files from smithi079:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/gabrioux-2024-09-21_11:23:40-orch:cephadm-wip-guits-main-2024-09-21-0842-distro-default-smithi/7913934/remote/smithi079 2024-09-21T12:38:23.124 DEBUG:teuthology.orchestra.run.smithi079:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-21T12:38:23.164 DEBUG:teuthology.misc:Transferring archived files from smithi104:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/gabrioux-2024-09-21_11:23:40-orch:cephadm-wip-guits-main-2024-09-21-0842-distro-default-smithi/7913934/remote/smithi104 2024-09-21T12:38:23.164 DEBUG:teuthology.orchestra.run.smithi104:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-21T12:38:23.200 INFO:teuthology.task.internal:Removing archive directory... 2024-09-21T12:38:23.200 DEBUG:teuthology.orchestra.run.smithi079:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-21T12:38:23.202 DEBUG:teuthology.orchestra.run.smithi104:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-21T12:38:23.249 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-21T12:38:23.265 INFO:teuthology.task.internal:Not uploading archives. 2024-09-21T12:38:23.265 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-21T12:38:23.304 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-21T12:38:23.304 DEBUG:teuthology.orchestra.run.smithi079:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-21T12:38:23.306 DEBUG:teuthology.orchestra.run.smithi104:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-21T12:38:23.318 INFO:teuthology.orchestra.run.smithi079.stdout: 265956 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 21 12:38 /home/ubuntu/cephtest 2024-09-21T12:38:23.320 INFO:teuthology.orchestra.run.smithi104.stdout: 265324 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 21 12:38 /home/ubuntu/cephtest 2024-09-21T12:38:23.322 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-21T12:38:23.331 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-21T12:38:23.352 INFO:teuthology.run:Summary data: description: orch:cephadm/upgrade/{1-start-distro/1-start-centos_9.stream-reef 2-repo_digest/defaut 3-upgrade/simple 4-wait 5-upgrade-ls agent/on mon_election/classic} duration: 1161.527646780014 failure_reason: 'Command failed on smithi079 with status 1: ''sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:reef shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 33168022-7814-11ef-baf6-efdc52797490 -e sha1=441784946648ff53dd310293f9dd752afeeea72a -- bash -c \''ceph versions | jq -e \''"\''"\''.overall | length == 1\''"\''"\''\''''' owner: scheduled_gabrioux@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=96bb007e133b49abbf635ac5cb77d7ff status: fail success: false 2024-09-21T12:38:23.352 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-21T12:38:23.418 INFO:teuthology.run:FAIL