2024-09-17T00:02:33.275 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-17T00:02:33.316 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-17T00:02:33.394 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-09-16_21:08:04-orch-squid-distro-default-smithi/7907924 branch: squid description: orch/cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/classic} email: ceph-qa@ceph.com first_in_suite: false job_id: '7907924' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: teuthology-2024-09-16_21:08:04-orch-squid-distro-default-smithi no_nested_subset: false os_type: centos os_version: 9.stream overrides: admin_socket: branch: squid ceph: conf: global: mon election default strategy: 1 mgr: debug mgr: 20 debug ms: 1 mgr/cephadm/use_agent: false mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - CEPHADM_STRAY_DAEMON - CEPHADM_FAILED_DAEMON - CEPHADM_AGENT_DOWN log-only-match: - CEPHADM_ sha1: c2ddf9721ef254645201cdbabd9c8f49c30a586f ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: c2ddf9721ef254645201cdbabd9c8f49c30a586f workunit: branch: squid sha1: e20bff7d16f5b09c58ead9548e8cf98cf87ee227 owner: scheduled_teuthology@teuthology priority: 100 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - grafana.a - node-exporter.b seed: 599 sha1: c2ddf9721ef254645201cdbabd9c8f49c30a586f sleep_before_teardown: 0 subset: 33/64 suite: orch suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_e20bff7d16f5b09c58ead9548e8cf98cf87ee227/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: e20bff7d16f5b09c58ead9548e8cf98cf87ee227 targets: smithi002.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHI3eHzsAywnEKRgEAFmulamVv2DKG8EQig6XswhQzFKV25VYhXJNcrTfdGaNnwYZTP1zTtfVZxCJMpdfvTAqT4= smithi099.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOeYugi4vePoXLupz5wzLd3j7aO+qwaKJJanOd+DvI5UpQ0hT4BNI6SR++A2gPcw/t2MFYRJe0muzyZi2mVI/y4= tasks: - cephadm: cephadm_branch: v17.2.0 cephadm_git_url: https://github.com/ceph/ceph image: quay.io/ceph/ceph:v17.2.0 - cephadm.shell: mon.a: - ceph config set mgr mgr/cephadm/use_repo_digest false --force - cephadm.shell: env: - sha1 mon.a: - radosgw-admin realm create --rgw-realm=r --default - radosgw-admin zonegroup create --rgw-zonegroup=default --master --default - radosgw-admin zone create --rgw-zonegroup=default --rgw-zone=z --master --default - radosgw-admin period update --rgw-realm=r --commit - ceph orch apply rgw foo --realm r --zone z --placement=2 --port=8000 - ceph orch apply rgw smpl - ceph osd pool create foo - rbd pool init foo - ceph orch apply iscsi foo u p - sleep 120 - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 mon.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done - ceph orch ps - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph orch upgrade status - ceph health detail - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 - ceph orch ls | grep '^osd ' - cephadm.shell: mon.a: - ceph orch upgrade ls - ceph orch upgrade ls --image quay.io/ceph/ceph --show-all-versions | grep 16.2.0 - ceph orch upgrade ls --image quay.io/ceph/ceph --tags | grep v16.2.2 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-16_21:08:04 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-17T00:02:33.395 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_e20bff7d16f5b09c58ead9548e8cf98cf87ee227/qa; will attempt to use it 2024-09-17T00:02:33.395 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_e20bff7d16f5b09c58ead9548e8cf98cf87ee227/qa/tasks 2024-09-17T00:02:33.395 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-17T00:02:33.397 INFO:teuthology.task.internal:Checking packages... 2024-09-17T00:02:33.417 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'c2ddf9721ef254645201cdbabd9c8f49c30a586f' 2024-09-17T00:02:33.417 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-17T00:02:33.417 INFO:teuthology.packaging:ref: None 2024-09-17T00:02:33.418 INFO:teuthology.packaging:tag: None 2024-09-17T00:02:33.418 INFO:teuthology.packaging:branch: squid 2024-09-17T00:02:33.418 INFO:teuthology.packaging:sha1: c2ddf9721ef254645201cdbabd9c8f49c30a586f 2024-09-17T00:02:33.418 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-09-17T00:02:33.649 INFO:teuthology.task.internal:Found packages for ceph version 19.2.0-0 2024-09-17T00:02:33.650 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-17T00:02:33.723 INFO:teuthology.task.internal:no buildpackages task found 2024-09-17T00:02:33.723 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-17T00:02:33.731 INFO:teuthology.task.internal:Saving configuration 2024-09-17T00:02:33.743 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-17T00:02:33.749 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-17T00:02:33.775 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi002.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-16_21:08:04-orch-squid-distro-default-smithi/7907924', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-16 23:57:35.603006', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHI3eHzsAywnEKRgEAFmulamVv2DKG8EQig6XswhQzFKV25VYhXJNcrTfdGaNnwYZTP1zTtfVZxCJMpdfvTAqT4='} 2024-09-17T00:02:33.795 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi099.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-16_21:08:04-orch-squid-distro-default-smithi/7907924', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-16 23:57:35.604614', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOeYugi4vePoXLupz5wzLd3j7aO+qwaKJJanOd+DvI5UpQ0hT4BNI6SR++A2gPcw/t2MFYRJe0muzyZi2mVI/y4='} 2024-09-17T00:02:33.795 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-17T00:02:33.801 INFO:teuthology.task.internal:roles: ubuntu@smithi002.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a', 'alertmanager.a'] 2024-09-17T00:02:33.801 INFO:teuthology.task.internal:roles: ubuntu@smithi099.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b'] 2024-09-17T00:02:33.801 INFO:teuthology.run_tasks:Running task console_log... 2024-09-17T00:02:33.859 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f42cfcfbac0>, signals=[15]) 2024-09-17T00:02:33.859 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-17T00:02:33.870 INFO:teuthology.task.internal:Opening connections... 2024-09-17T00:02:33.871 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi002.front.sepia.ceph.com 2024-09-17T00:02:33.872 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi002.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T00:02:33.951 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi099.front.sepia.ceph.com 2024-09-17T00:02:33.952 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi099.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T00:02:34.026 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-17T00:02:34.052 DEBUG:teuthology.orchestra.run.smithi002:> uname -m 2024-09-17T00:02:34.071 INFO:teuthology.orchestra.run.smithi002.stdout:x86_64 2024-09-17T00:02:34.071 DEBUG:teuthology.orchestra.run.smithi002:> cat /etc/os-release 2024-09-17T00:02:34.128 INFO:teuthology.orchestra.run.smithi002.stdout:NAME="CentOS Stream" 2024-09-17T00:02:34.128 INFO:teuthology.orchestra.run.smithi002.stdout:VERSION="9" 2024-09-17T00:02:34.128 INFO:teuthology.orchestra.run.smithi002.stdout:ID="centos" 2024-09-17T00:02:34.128 INFO:teuthology.orchestra.run.smithi002.stdout:ID_LIKE="rhel fedora" 2024-09-17T00:02:34.128 INFO:teuthology.orchestra.run.smithi002.stdout:VERSION_ID="9" 2024-09-17T00:02:34.129 INFO:teuthology.orchestra.run.smithi002.stdout:PLATFORM_ID="platform:el9" 2024-09-17T00:02:34.129 INFO:teuthology.orchestra.run.smithi002.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-17T00:02:34.129 INFO:teuthology.orchestra.run.smithi002.stdout:ANSI_COLOR="0;31" 2024-09-17T00:02:34.129 INFO:teuthology.orchestra.run.smithi002.stdout:LOGO="fedora-logo-icon" 2024-09-17T00:02:34.129 INFO:teuthology.orchestra.run.smithi002.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-17T00:02:34.129 INFO:teuthology.orchestra.run.smithi002.stdout:HOME_URL="https://centos.org/" 2024-09-17T00:02:34.129 INFO:teuthology.orchestra.run.smithi002.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-17T00:02:34.129 INFO:teuthology.orchestra.run.smithi002.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-17T00:02:34.129 INFO:teuthology.orchestra.run.smithi002.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-17T00:02:34.130 INFO:teuthology.lock.ops:Updating smithi002.front.sepia.ceph.com on lock server 2024-09-17T00:02:34.148 DEBUG:teuthology.orchestra.run.smithi099:> uname -m 2024-09-17T00:02:34.166 INFO:teuthology.orchestra.run.smithi099.stdout:x86_64 2024-09-17T00:02:34.166 DEBUG:teuthology.orchestra.run.smithi099:> cat /etc/os-release 2024-09-17T00:02:34.223 INFO:teuthology.orchestra.run.smithi099.stdout:NAME="CentOS Stream" 2024-09-17T00:02:34.223 INFO:teuthology.orchestra.run.smithi099.stdout:VERSION="9" 2024-09-17T00:02:34.223 INFO:teuthology.orchestra.run.smithi099.stdout:ID="centos" 2024-09-17T00:02:34.223 INFO:teuthology.orchestra.run.smithi099.stdout:ID_LIKE="rhel fedora" 2024-09-17T00:02:34.223 INFO:teuthology.orchestra.run.smithi099.stdout:VERSION_ID="9" 2024-09-17T00:02:34.223 INFO:teuthology.orchestra.run.smithi099.stdout:PLATFORM_ID="platform:el9" 2024-09-17T00:02:34.223 INFO:teuthology.orchestra.run.smithi099.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-17T00:02:34.223 INFO:teuthology.orchestra.run.smithi099.stdout:ANSI_COLOR="0;31" 2024-09-17T00:02:34.223 INFO:teuthology.orchestra.run.smithi099.stdout:LOGO="fedora-logo-icon" 2024-09-17T00:02:34.223 INFO:teuthology.orchestra.run.smithi099.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-17T00:02:34.223 INFO:teuthology.orchestra.run.smithi099.stdout:HOME_URL="https://centos.org/" 2024-09-17T00:02:34.224 INFO:teuthology.orchestra.run.smithi099.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-17T00:02:34.224 INFO:teuthology.orchestra.run.smithi099.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-17T00:02:34.224 INFO:teuthology.orchestra.run.smithi099.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-17T00:02:34.224 INFO:teuthology.lock.ops:Updating smithi099.front.sepia.ceph.com on lock server 2024-09-17T00:02:34.247 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-17T00:02:34.256 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-17T00:02:34.261 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-17T00:02:34.262 DEBUG:teuthology.orchestra.run.smithi002:> test '!' -e /home/ubuntu/cephtest 2024-09-17T00:02:34.264 DEBUG:teuthology.orchestra.run.smithi099:> test '!' -e /home/ubuntu/cephtest 2024-09-17T00:02:34.280 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-17T00:02:34.288 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-17T00:02:34.289 DEBUG:teuthology.orchestra.run.smithi002:> test -z $(ls -A /var/lib/ceph) 2024-09-17T00:02:34.322 DEBUG:teuthology.orchestra.run.smithi099:> test -z $(ls -A /var/lib/ceph) 2024-09-17T00:02:34.348 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-17T00:02:34.402 INFO:teuthology.run_tasks:Running task kernel... 2024-09-17T00:02:34.415 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-17T00:02:34.415 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-17T00:02:34.415 DEBUG:teuthology.orchestra.run.smithi002:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-17T00:02:34.416 DEBUG:teuthology.orchestra.run.smithi099:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-17T00:02:34.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T00:02:34.432 DEBUG:teuthology.orchestra.run.smithi099:> uname -r 2024-09-17T00:02:34.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T00:02:34.433 DEBUG:teuthology.orchestra.run.smithi002:> uname -r 2024-09-17T00:02:34.489 INFO:teuthology.orchestra.run.smithi099.stdout:5.14.0-503.el9.x86_64 2024-09-17T00:02:34.489 INFO:teuthology.task.kernel:Running kernel on smithi099: 5.14.0-503.el9.x86_64 2024-09-17T00:02:34.489 DEBUG:teuthology.orchestra.run.smithi099:> sudo yum install -y kernel 2024-09-17T00:02:34.495 INFO:teuthology.orchestra.run.smithi002.stdout:5.14.0-503.el9.x86_64 2024-09-17T00:02:34.495 INFO:teuthology.task.kernel:Running kernel on smithi002: 5.14.0-503.el9.x86_64 2024-09-17T00:02:34.495 DEBUG:teuthology.orchestra.run.smithi002:> sudo yum install -y kernel 2024-09-17T00:02:40.453 INFO:teuthology.orchestra.run.smithi002.stdout:CentOS Stream 9 - BaseOS 2.0 MB/s | 8.3 MB 00:04 2024-09-17T00:02:42.851 INFO:teuthology.orchestra.run.smithi099.stdout:CentOS Stream 9 - BaseOS 1.2 MB/s | 8.3 MB 00:06 2024-09-17T00:02:45.752 INFO:teuthology.orchestra.run.smithi099.stdout:CentOS Stream 9 - AppStream 11 MB/s | 20 MB 00:01 2024-09-17T00:02:45.962 INFO:teuthology.orchestra.run.smithi002.stdout:CentOS Stream 9 - AppStream 4.6 MB/s | 20 MB 00:04 2024-09-17T00:02:50.187 INFO:teuthology.orchestra.run.smithi099.stdout:CentOS Stream 9 - CRB 9.7 MB/s | 6.5 MB 00:00 2024-09-17T00:02:51.165 INFO:teuthology.orchestra.run.smithi002.stdout:CentOS Stream 9 - CRB 5.4 MB/s | 6.5 MB 00:01 2024-09-17T00:02:52.587 INFO:teuthology.orchestra.run.smithi099.stdout:CentOS Stream 9 - Extras packages 16 kB/s | 19 kB 00:01 2024-09-17T00:02:52.814 INFO:teuthology.orchestra.run.smithi002.stdout:CentOS Stream 9 - Extras packages 52 kB/s | 19 kB 00:00 2024-09-17T00:02:53.374 INFO:teuthology.orchestra.run.smithi099.stdout:Extra Packages for Enterprise Linux 34 MB/s | 23 MB 00:00 2024-09-17T00:02:53.984 INFO:teuthology.orchestra.run.smithi002.stdout:Extra Packages for Enterprise Linux 22 MB/s | 23 MB 00:01 2024-09-17T00:02:58.754 INFO:teuthology.orchestra.run.smithi099.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-09-17T00:02:59.590 INFO:teuthology.orchestra.run.smithi002.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-09-17T00:03:00.669 INFO:teuthology.orchestra.run.smithi099.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-17T00:03:00.669 INFO:teuthology.orchestra.run.smithi099.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-17T00:03:00.754 INFO:teuthology.orchestra.run.smithi099.stdout:Dependencies resolved. 2024-09-17T00:03:00.761 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-09-17T00:03:00.762 INFO:teuthology.orchestra.run.smithi099.stdout: Package Architecture Version Repository Size 2024-09-17T00:03:00.762 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-09-17T00:03:00.762 INFO:teuthology.orchestra.run.smithi099.stdout:Installing: 2024-09-17T00:03:00.762 INFO:teuthology.orchestra.run.smithi099.stdout: kernel x86_64 5.14.0-508.el9 baseos 36 k 2024-09-17T00:03:00.762 INFO:teuthology.orchestra.run.smithi099.stdout:Installing dependencies: 2024-09-17T00:03:00.762 INFO:teuthology.orchestra.run.smithi099.stdout: kernel-core x86_64 5.14.0-508.el9 baseos 16 M 2024-09-17T00:03:00.763 INFO:teuthology.orchestra.run.smithi099.stdout: kernel-modules x86_64 5.14.0-508.el9 baseos 35 M 2024-09-17T00:03:00.763 INFO:teuthology.orchestra.run.smithi099.stdout: kernel-modules-core x86_64 5.14.0-508.el9 baseos 29 M 2024-09-17T00:03:00.763 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-09-17T00:03:00.763 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction Summary 2024-09-17T00:03:00.763 INFO:teuthology.orchestra.run.smithi099.stdout:================================================================================ 2024-09-17T00:03:00.763 INFO:teuthology.orchestra.run.smithi099.stdout:Install 4 Packages 2024-09-17T00:03:00.763 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-09-17T00:03:00.763 INFO:teuthology.orchestra.run.smithi099.stdout:Total download size: 79 M 2024-09-17T00:03:00.764 INFO:teuthology.orchestra.run.smithi099.stdout:Installed size: 126 M 2024-09-17T00:03:00.764 INFO:teuthology.orchestra.run.smithi099.stdout:Downloading Packages: 2024-09-17T00:03:01.114 INFO:teuthology.orchestra.run.smithi099.stdout:(1/4): kernel-5.14.0-508.el9.x86_64.rpm 120 kB/s | 36 kB 00:00 2024-09-17T00:03:01.547 INFO:teuthology.orchestra.run.smithi002.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-17T00:03:01.547 INFO:teuthology.orchestra.run.smithi002.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-17T00:03:01.636 INFO:teuthology.orchestra.run.smithi002.stdout:Dependencies resolved. 2024-09-17T00:03:01.651 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2024-09-17T00:03:01.651 INFO:teuthology.orchestra.run.smithi002.stdout: Package Architecture Version Repository Size 2024-09-17T00:03:01.652 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2024-09-17T00:03:01.652 INFO:teuthology.orchestra.run.smithi002.stdout:Installing: 2024-09-17T00:03:01.652 INFO:teuthology.orchestra.run.smithi002.stdout: kernel x86_64 5.14.0-508.el9 baseos 36 k 2024-09-17T00:03:01.652 INFO:teuthology.orchestra.run.smithi002.stdout:Installing dependencies: 2024-09-17T00:03:01.652 INFO:teuthology.orchestra.run.smithi002.stdout: kernel-core x86_64 5.14.0-508.el9 baseos 16 M 2024-09-17T00:03:01.652 INFO:teuthology.orchestra.run.smithi002.stdout: kernel-modules x86_64 5.14.0-508.el9 baseos 35 M 2024-09-17T00:03:01.652 INFO:teuthology.orchestra.run.smithi002.stdout: kernel-modules-core x86_64 5.14.0-508.el9 baseos 29 M 2024-09-17T00:03:01.652 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-17T00:03:01.653 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction Summary 2024-09-17T00:03:01.653 INFO:teuthology.orchestra.run.smithi002.stdout:================================================================================ 2024-09-17T00:03:01.653 INFO:teuthology.orchestra.run.smithi002.stdout:Install 4 Packages 2024-09-17T00:03:01.653 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-17T00:03:01.653 INFO:teuthology.orchestra.run.smithi002.stdout:Total download size: 79 M 2024-09-17T00:03:01.653 INFO:teuthology.orchestra.run.smithi002.stdout:Installed size: 126 M 2024-09-17T00:03:01.654 INFO:teuthology.orchestra.run.smithi002.stdout:Downloading Packages: 2024-09-17T00:03:01.808 INFO:teuthology.orchestra.run.smithi002.stdout:(1/4): kernel-5.14.0-508.el9.x86_64.rpm 300 kB/s | 36 kB 00:00 2024-09-17T00:03:08.774 INFO:teuthology.orchestra.run.smithi002.stdout:(2/4): kernel-core-5.14.0-508.el9.x86_64.rpm 2.2 MB/s | 16 MB 00:07 2024-09-17T00:03:14.816 INFO:teuthology.orchestra.run.smithi002.stdout:(3/4): kernel-modules-core-5.14.0-508.el9.x86_6 2.2 MB/s | 29 MB 00:13 2024-09-17T00:03:16.612 INFO:teuthology.orchestra.run.smithi099.stdout:(2/4): kernel-core-5.14.0-508.el9.x86_64.rpm 1.0 MB/s | 16 MB 00:15 2024-09-17T00:03:17.224 INFO:teuthology.orchestra.run.smithi002.stdout:(4/4): kernel-modules-5.14.0-508.el9.x86_64.rpm 2.2 MB/s | 35 MB 00:15 2024-09-17T00:03:17.224 INFO:teuthology.orchestra.run.smithi002.stdout:-------------------------------------------------------------------------------- 2024-09-17T00:03:17.224 INFO:teuthology.orchestra.run.smithi002.stdout:Total 5.1 MB/s | 79 MB 00:15 2024-09-17T00:03:17.935 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction check 2024-09-17T00:03:18.318 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction check succeeded. 2024-09-17T00:03:18.319 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction test 2024-09-17T00:03:19.292 INFO:teuthology.orchestra.run.smithi002.stdout:Transaction test succeeded. 2024-09-17T00:03:19.293 INFO:teuthology.orchestra.run.smithi002.stdout:Running transaction 2024-09-17T00:03:21.861 INFO:teuthology.orchestra.run.smithi002.stdout: Preparing : 1/1 2024-09-17T00:03:22.552 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : kernel-modules-core-5.14.0-508.el9.x86_64 1/4 2024-09-17T00:03:22.632 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-17T00:03:25.383 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-17T00:03:25.510 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-17T00:03:30.637 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-17T00:03:30.678 INFO:teuthology.orchestra.run.smithi002.stdout: Installing : kernel-5.14.0-508.el9.x86_64 4/4 2024-09-17T00:03:35.812 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-17T00:03:36.708 INFO:teuthology.orchestra.run.smithi099.stdout:(3/4): kernel-modules-core-5.14.0-508.el9.x86_6 822 kB/s | 29 MB 00:35 2024-09-17T00:03:42.749 INFO:teuthology.orchestra.run.smithi099.stdout:(4/4): kernel-modules-5.14.0-508.el9.x86_64.rpm 843 kB/s | 35 MB 00:41 2024-09-17T00:03:42.749 INFO:teuthology.orchestra.run.smithi099.stdout:-------------------------------------------------------------------------------- 2024-09-17T00:03:42.749 INFO:teuthology.orchestra.run.smithi099.stdout:Total 1.9 MB/s | 79 MB 00:41 2024-09-17T00:03:43.428 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction check 2024-09-17T00:03:43.785 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction check succeeded. 2024-09-17T00:03:43.785 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction test 2024-09-17T00:03:44.736 INFO:teuthology.orchestra.run.smithi099.stdout:Transaction test succeeded. 2024-09-17T00:03:44.736 INFO:teuthology.orchestra.run.smithi099.stdout:Running transaction 2024-09-17T00:03:47.167 INFO:teuthology.orchestra.run.smithi099.stdout: Preparing : 1/1 2024-09-17T00:03:47.814 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : kernel-modules-core-5.14.0-508.el9.x86_64 1/4 2024-09-17T00:03:47.894 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-17T00:03:50.657 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-17T00:03:50.806 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-17T00:03:55.660 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-17T00:03:55.755 INFO:teuthology.orchestra.run.smithi099.stdout: Installing : kernel-5.14.0-508.el9.x86_64 4/4 2024-09-17T00:04:00.587 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-17T00:04:14.629 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 4/4 2024-09-17T00:04:14.669 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 4/4 2024-09-17T00:04:15.043 INFO:teuthology.orchestra.run.smithi002.stdout: Running scriptlet: kernel-5.14.0-508.el9.x86_64 4/4 2024-09-17T00:04:15.043 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : kernel-5.14.0-508.el9.x86_64 1/4 2024-09-17T00:04:15.043 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-17T00:04:15.043 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-17T00:04:15.907 INFO:teuthology.orchestra.run.smithi002.stdout: Verifying : kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-17T00:04:15.907 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-17T00:04:15.907 INFO:teuthology.orchestra.run.smithi002.stdout:Installed: 2024-09-17T00:04:15.907 INFO:teuthology.orchestra.run.smithi002.stdout: kernel-5.14.0-508.el9.x86_64 2024-09-17T00:04:15.907 INFO:teuthology.orchestra.run.smithi002.stdout: kernel-core-5.14.0-508.el9.x86_64 2024-09-17T00:04:15.907 INFO:teuthology.orchestra.run.smithi002.stdout: kernel-modules-5.14.0-508.el9.x86_64 2024-09-17T00:04:15.907 INFO:teuthology.orchestra.run.smithi002.stdout: kernel-modules-core-5.14.0-508.el9.x86_64 2024-09-17T00:04:15.908 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-17T00:04:15.908 INFO:teuthology.orchestra.run.smithi002.stdout:Complete! 2024-09-17T00:04:16.250 DEBUG:teuthology.orchestra.run.smithi002:> rpm -q kernel | sort -rV | head -n 1 2024-09-17T00:04:16.343 INFO:teuthology.orchestra.run.smithi002.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-17T00:04:16.344 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-17T00:04:16.344 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-508.el9.x86_64 2024-09-17T00:04:16.344 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-17T00:04:16.344 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-17T00:04:16.344 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-17T00:04:16.344 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi002.front.sepia.ceph.com, path=None, version=distro) 2024-09-17T00:04:16.345 DEBUG:teuthology.orchestra.run.smithi002:> rpm -q kernel | sort -rV | head -n 1 2024-09-17T00:04:16.403 INFO:teuthology.orchestra.run.smithi002.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-17T00:04:16.403 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-17T00:04:16.404 DEBUG:teuthology.orchestra.run.smithi002:> sudo rpm -qi grub2-tools 2024-09-17T00:04:16.439 INFO:teuthology.orchestra.run.smithi002.stdout:Name : grub2-tools 2024-09-17T00:04:16.439 INFO:teuthology.orchestra.run.smithi002.stdout:Epoch : 1 2024-09-17T00:04:16.439 INFO:teuthology.orchestra.run.smithi002.stdout:Version : 2.06 2024-09-17T00:04:16.439 INFO:teuthology.orchestra.run.smithi002.stdout:Release : 82.el9 2024-09-17T00:04:16.439 INFO:teuthology.orchestra.run.smithi002.stdout:Architecture: x86_64 2024-09-17T00:04:16.440 INFO:teuthology.orchestra.run.smithi002.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-17T00:04:16.440 INFO:teuthology.orchestra.run.smithi002.stdout:Group : Unspecified 2024-09-17T00:04:16.440 INFO:teuthology.orchestra.run.smithi002.stdout:Size : 8274670 2024-09-17T00:04:16.440 INFO:teuthology.orchestra.run.smithi002.stdout:License : GPLv3+ 2024-09-17T00:04:16.440 INFO:teuthology.orchestra.run.smithi002.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-17T00:04:16.440 INFO:teuthology.orchestra.run.smithi002.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-17T00:04:16.440 INFO:teuthology.orchestra.run.smithi002.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-17T00:04:16.440 INFO:teuthology.orchestra.run.smithi002.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-17T00:04:16.440 INFO:teuthology.orchestra.run.smithi002.stdout:Packager : builder@centos.org 2024-09-17T00:04:16.440 INFO:teuthology.orchestra.run.smithi002.stdout:Vendor : CentOS 2024-09-17T00:04:16.440 INFO:teuthology.orchestra.run.smithi002.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-17T00:04:16.440 INFO:teuthology.orchestra.run.smithi002.stdout:Summary : Support tools for GRUB. 2024-09-17T00:04:16.440 INFO:teuthology.orchestra.run.smithi002.stdout:Description : 2024-09-17T00:04:16.441 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-17T00:04:16.441 INFO:teuthology.orchestra.run.smithi002.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-17T00:04:16.441 INFO:teuthology.orchestra.run.smithi002.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-17T00:04:16.441 INFO:teuthology.orchestra.run.smithi002.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-17T00:04:16.441 INFO:teuthology.orchestra.run.smithi002.stdout:hardware devices. 2024-09-17T00:04:16.441 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-17T00:04:16.441 INFO:teuthology.orchestra.run.smithi002.stdout:This subpackage provides tools for support of all platforms. 2024-09-17T00:04:16.442 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-17T00:04:16.442 INFO:teuthology.task.kernel:Updating grub on smithi002 to boot 5.14.0-508.el9.x86_64 2024-09-17T00:04:16.442 DEBUG:teuthology.orchestra.run.smithi002:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-17T00:04:17.173 INFO:teuthology.orchestra.run.smithi002.stderr:Generating grub configuration file ... 2024-09-17T00:04:18.657 INFO:teuthology.orchestra.run.smithi002.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-17T00:04:18.699 INFO:teuthology.orchestra.run.smithi002.stderr:done 2024-09-17T00:04:18.701 DEBUG:teuthology.orchestra.run.smithi002:> mktemp 2024-09-17T00:04:18.716 INFO:teuthology.orchestra.run.smithi002.stdout:/tmp/tmp.4FhwTOKsY4 2024-09-17T00:04:18.717 DEBUG:teuthology.orchestra.run.smithi002:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.4FhwTOKsY4 2024-09-17T00:04:18.783 DEBUG:teuthology.orchestra.run.smithi002:> sudo chmod 0666 /tmp/tmp.4FhwTOKsY4 2024-09-17T00:04:18.920 DEBUG:teuthology.orchestra.remote:smithi002:/tmp/tmp.4FhwTOKsY4 is 6KB 2024-09-17T00:04:18.969 DEBUG:teuthology.orchestra.run.smithi002:> rm -fr /tmp/tmp.4FhwTOKsY4 2024-09-17T00:04:18.984 DEBUG:teuthology.orchestra.run.smithi002:> sudo /bin/ls /boot/loader/entries || true 2024-09-17T00:04:19.047 INFO:teuthology.orchestra.run.smithi002.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-17T00:04:19.047 INFO:teuthology.orchestra.run.smithi002.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-17T00:04:19.047 INFO:teuthology.orchestra.run.smithi002.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-17T00:04:19.047 INFO:teuthology.orchestra.run.smithi002.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64.conf 2024-09-17T00:04:19.048 DEBUG:teuthology.orchestra.run.smithi002:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64 2024-09-17T00:04:19.202 DEBUG:teuthology.orchestra.run.smithi002:> sudo shutdown -r now 2024-09-17T00:04:38.678 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 4/4 2024-09-17T00:04:38.717 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 4/4 2024-09-17T00:04:39.113 INFO:teuthology.orchestra.run.smithi099.stdout: Running scriptlet: kernel-5.14.0-508.el9.x86_64 4/4 2024-09-17T00:04:39.113 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : kernel-5.14.0-508.el9.x86_64 1/4 2024-09-17T00:04:39.113 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-17T00:04:39.113 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-17T00:04:39.909 INFO:teuthology.orchestra.run.smithi099.stdout: Verifying : kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-17T00:04:39.909 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-09-17T00:04:39.909 INFO:teuthology.orchestra.run.smithi099.stdout:Installed: 2024-09-17T00:04:39.910 INFO:teuthology.orchestra.run.smithi099.stdout: kernel-5.14.0-508.el9.x86_64 2024-09-17T00:04:39.910 INFO:teuthology.orchestra.run.smithi099.stdout: kernel-core-5.14.0-508.el9.x86_64 2024-09-17T00:04:39.910 INFO:teuthology.orchestra.run.smithi099.stdout: kernel-modules-5.14.0-508.el9.x86_64 2024-09-17T00:04:39.910 INFO:teuthology.orchestra.run.smithi099.stdout: kernel-modules-core-5.14.0-508.el9.x86_64 2024-09-17T00:04:39.910 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-09-17T00:04:39.910 INFO:teuthology.orchestra.run.smithi099.stdout:Complete! 2024-09-17T00:04:40.257 DEBUG:teuthology.orchestra.run.smithi099:> rpm -q kernel | sort -rV | head -n 1 2024-09-17T00:04:40.345 INFO:teuthology.orchestra.run.smithi099.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-17T00:04:40.345 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-17T00:04:40.346 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-508.el9.x86_64 2024-09-17T00:04:40.346 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-17T00:04:40.346 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-17T00:04:40.346 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-09-17T00:04:40.346 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi099.front.sepia.ceph.com, path=None, version=distro) 2024-09-17T00:04:40.346 DEBUG:teuthology.orchestra.run.smithi099:> rpm -q kernel | sort -rV | head -n 1 2024-09-17T00:04:40.405 INFO:teuthology.orchestra.run.smithi099.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-17T00:04:40.406 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-17T00:04:40.406 DEBUG:teuthology.orchestra.run.smithi099:> sudo rpm -qi grub2-tools 2024-09-17T00:04:40.447 INFO:teuthology.orchestra.run.smithi099.stdout:Name : grub2-tools 2024-09-17T00:04:40.447 INFO:teuthology.orchestra.run.smithi099.stdout:Epoch : 1 2024-09-17T00:04:40.447 INFO:teuthology.orchestra.run.smithi099.stdout:Version : 2.06 2024-09-17T00:04:40.448 INFO:teuthology.orchestra.run.smithi099.stdout:Release : 82.el9 2024-09-17T00:04:40.448 INFO:teuthology.orchestra.run.smithi099.stdout:Architecture: x86_64 2024-09-17T00:04:40.448 INFO:teuthology.orchestra.run.smithi099.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-17T00:04:40.448 INFO:teuthology.orchestra.run.smithi099.stdout:Group : Unspecified 2024-09-17T00:04:40.448 INFO:teuthology.orchestra.run.smithi099.stdout:Size : 8274670 2024-09-17T00:04:40.448 INFO:teuthology.orchestra.run.smithi099.stdout:License : GPLv3+ 2024-09-17T00:04:40.448 INFO:teuthology.orchestra.run.smithi099.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-17T00:04:40.448 INFO:teuthology.orchestra.run.smithi099.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-17T00:04:40.448 INFO:teuthology.orchestra.run.smithi099.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-17T00:04:40.448 INFO:teuthology.orchestra.run.smithi099.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-17T00:04:40.448 INFO:teuthology.orchestra.run.smithi099.stdout:Packager : builder@centos.org 2024-09-17T00:04:40.448 INFO:teuthology.orchestra.run.smithi099.stdout:Vendor : CentOS 2024-09-17T00:04:40.448 INFO:teuthology.orchestra.run.smithi099.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-17T00:04:40.449 INFO:teuthology.orchestra.run.smithi099.stdout:Summary : Support tools for GRUB. 2024-09-17T00:04:40.449 INFO:teuthology.orchestra.run.smithi099.stdout:Description : 2024-09-17T00:04:40.449 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-09-17T00:04:40.449 INFO:teuthology.orchestra.run.smithi099.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-17T00:04:40.449 INFO:teuthology.orchestra.run.smithi099.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-17T00:04:40.449 INFO:teuthology.orchestra.run.smithi099.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-17T00:04:40.449 INFO:teuthology.orchestra.run.smithi099.stdout:hardware devices. 2024-09-17T00:04:40.449 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-09-17T00:04:40.449 INFO:teuthology.orchestra.run.smithi099.stdout:This subpackage provides tools for support of all platforms. 2024-09-17T00:04:40.450 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-17T00:04:40.450 INFO:teuthology.task.kernel:Updating grub on smithi099 to boot 5.14.0-508.el9.x86_64 2024-09-17T00:04:40.450 DEBUG:teuthology.orchestra.run.smithi099:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-17T00:04:41.112 INFO:teuthology.orchestra.run.smithi099.stderr:Generating grub configuration file ... 2024-09-17T00:04:42.460 INFO:teuthology.orchestra.run.smithi099.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-17T00:04:42.485 INFO:teuthology.orchestra.run.smithi099.stderr:done 2024-09-17T00:04:42.487 DEBUG:teuthology.orchestra.run.smithi099:> mktemp 2024-09-17T00:04:42.503 INFO:teuthology.orchestra.run.smithi099.stdout:/tmp/tmp.Z6BQixEu02 2024-09-17T00:04:42.503 DEBUG:teuthology.orchestra.run.smithi099:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.Z6BQixEu02 2024-09-17T00:04:42.570 DEBUG:teuthology.orchestra.run.smithi099:> sudo chmod 0666 /tmp/tmp.Z6BQixEu02 2024-09-17T00:04:42.712 DEBUG:teuthology.orchestra.remote:smithi099:/tmp/tmp.Z6BQixEu02 is 6KB 2024-09-17T00:04:42.761 DEBUG:teuthology.orchestra.run.smithi099:> rm -fr /tmp/tmp.Z6BQixEu02 2024-09-17T00:04:42.776 DEBUG:teuthology.orchestra.run.smithi099:> sudo /bin/ls /boot/loader/entries || true 2024-09-17T00:04:42.838 INFO:teuthology.orchestra.run.smithi099.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-17T00:04:42.838 INFO:teuthology.orchestra.run.smithi099.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-17T00:04:42.838 INFO:teuthology.orchestra.run.smithi099.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-17T00:04:42.838 INFO:teuthology.orchestra.run.smithi099.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64.conf 2024-09-17T00:04:42.840 DEBUG:teuthology.orchestra.run.smithi099:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64 2024-09-17T00:04:42.997 DEBUG:teuthology.orchestra.run.smithi099:> sudo shutdown -r now 2024-09-17T00:04:49.209 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-17T00:04:49.209 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi002.front.sepia.ceph.com' 2024-09-17T00:04:49.210 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi002.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T00:05:07.773 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.2 2024-09-17T00:05:13.005 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-09-17T00:05:13.051 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi099.front.sepia.ceph.com' 2024-09-17T00:05:13.052 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi099.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T00:05:16.777 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi002.front.sepia.ceph.com' 2024-09-17T00:05:16.777 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi002.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T00:06:13.095 DEBUG:teuthology.orchestra.remote:timed out 2024-09-17T00:06:16.782 DEBUG:teuthology.orchestra.remote:timed out 2024-09-17T00:06:22.101 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi099.front.sepia.ceph.com' 2024-09-17T00:06:22.103 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi099.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T00:06:22.484 DEBUG:teuthology.orchestra.run.smithi099:> true 2024-09-17T00:06:22.843 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi099.front.sepia.ceph.com' 2024-09-17T00:06:22.843 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-508.el9.x86_64"... 2024-09-17T00:06:22.843 DEBUG:teuthology.orchestra.run.smithi099:> uname -r 2024-09-17T00:06:22.858 INFO:teuthology.orchestra.run.smithi099.stdout:5.14.0-508.el9.x86_64 2024-09-17T00:06:22.858 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-508.el9.x86_64 vs 5.14.0-508.el9.x86_64 2024-09-17T00:06:22.858 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-17T00:06:22.858 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-17T00:06:23.859 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-09-17T00:06:23.860 DEBUG:teuthology.orchestra.run.smithi099:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-17T00:06:23.955 INFO:teuthology.orchestra.run.smithi099.stdout:ttyS1 2024-09-17T00:06:23.982 DEBUG:teuthology.parallel:result is None 2024-09-17T00:06:28.783 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi002.front.sepia.ceph.com' 2024-09-17T00:06:28.783 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi002.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T00:06:29.121 DEBUG:teuthology.orchestra.run.smithi002:> true 2024-09-17T00:06:29.492 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi002.front.sepia.ceph.com' 2024-09-17T00:06:29.492 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-508.el9.x86_64"... 2024-09-17T00:06:29.492 DEBUG:teuthology.orchestra.run.smithi002:> uname -r 2024-09-17T00:06:29.509 INFO:teuthology.orchestra.run.smithi002.stdout:5.14.0-508.el9.x86_64 2024-09-17T00:06:29.509 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-508.el9.x86_64 vs 5.14.0-508.el9.x86_64 2024-09-17T00:06:29.509 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-17T00:06:29.509 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-17T00:06:30.510 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-17T00:06:30.510 DEBUG:teuthology.orchestra.run.smithi002:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-17T00:06:30.602 INFO:teuthology.orchestra.run.smithi002.stdout:ttyS1 2024-09-17T00:06:30.621 DEBUG:teuthology.parallel:result is None 2024-09-17T00:06:30.621 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-17T00:06:30.628 INFO:teuthology.task.internal:Creating test directory... 2024-09-17T00:06:30.628 DEBUG:teuthology.orchestra.run.smithi002:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-17T00:06:30.631 DEBUG:teuthology.orchestra.run.smithi099:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-17T00:06:30.651 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-17T00:06:30.656 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-17T00:06:30.660 INFO:teuthology.task.internal:Creating archive directory... 2024-09-17T00:06:30.660 DEBUG:teuthology.orchestra.run.smithi002:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-17T00:06:30.688 DEBUG:teuthology.orchestra.run.smithi099:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-17T00:06:30.793 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-17T00:06:30.799 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-17T00:06:30.799 DEBUG:teuthology.orchestra.run.smithi002:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-17T00:06:30.802 DEBUG:teuthology.orchestra.run.smithi099:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-17T00:06:30.830 INFO:teuthology.orchestra.run.smithi099.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T00:06:30.837 INFO:teuthology.orchestra.run.smithi002.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T00:06:30.841 INFO:teuthology.orchestra.run.smithi099.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T00:06:30.849 INFO:teuthology.orchestra.run.smithi002.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-17T00:06:30.851 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-17T00:06:30.869 INFO:teuthology.task.internal:Configuring sudo... 2024-09-17T00:06:30.869 DEBUG:teuthology.orchestra.run.smithi002:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-17T00:06:30.893 DEBUG:teuthology.orchestra.run.smithi099:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-17T00:06:30.920 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-17T00:06:30.942 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-17T00:06:30.942 DEBUG:teuthology.orchestra.run.smithi002:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-17T00:06:30.963 DEBUG:teuthology.orchestra.run.smithi099:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-17T00:06:30.980 DEBUG:teuthology.orchestra.run.smithi002:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-17T00:06:31.065 DEBUG:teuthology.orchestra.run.smithi002:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-17T00:06:31.112 DEBUG:teuthology.orchestra.run.smithi002:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-17T00:06:31.196 DEBUG:teuthology.orchestra.run.smithi002:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-17T00:06:31.238 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-09-17T00:06:31.238 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-17T00:06:31.305 DEBUG:teuthology.orchestra.run.smithi099:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-17T00:06:31.348 DEBUG:teuthology.orchestra.run.smithi099:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-17T00:06:31.411 DEBUG:teuthology.orchestra.run.smithi099:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-17T00:06:31.500 DEBUG:teuthology.orchestra.run.smithi099:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-17T00:06:31.544 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-09-17T00:06:31.544 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-17T00:06:31.613 DEBUG:teuthology.orchestra.run.smithi002:> sudo service rsyslog restart 2024-09-17T00:06:31.616 DEBUG:teuthology.orchestra.run.smithi099:> sudo service rsyslog restart 2024-09-17T00:06:31.655 INFO:teuthology.orchestra.run.smithi002.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-17T00:06:31.707 INFO:teuthology.orchestra.run.smithi099.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-17T00:06:32.253 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-17T00:06:32.262 INFO:teuthology.task.internal:Starting timer... 2024-09-17T00:06:32.262 INFO:teuthology.run_tasks:Running task pcp... 2024-09-17T00:06:32.274 INFO:teuthology.run_tasks:Running task selinux... 2024-09-17T00:06:32.286 DEBUG:teuthology.orchestra.run.smithi002:> sudo service auditd rotate 2024-09-17T00:06:32.331 INFO:teuthology.orchestra.run.smithi002.stdout:Rotating logs: 2024-09-17T00:06:32.333 DEBUG:teuthology.orchestra.run.smithi099:> sudo service auditd rotate 2024-09-17T00:06:32.390 INFO:teuthology.orchestra.run.smithi099.stdout:Rotating logs: 2024-09-17T00:06:32.391 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-17T00:06:32.392 DEBUG:teuthology.orchestra.run.smithi002:> /usr/sbin/getenforce 2024-09-17T00:06:32.420 INFO:teuthology.orchestra.run.smithi002.stdout:Permissive 2024-09-17T00:06:32.420 DEBUG:teuthology.orchestra.run.smithi099:> /usr/sbin/getenforce 2024-09-17T00:06:32.449 INFO:teuthology.orchestra.run.smithi099.stdout:Permissive 2024-09-17T00:06:32.449 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi002.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi099.front.sepia.ceph.com': 'permissive'} 2024-09-17T00:06:32.450 DEBUG:teuthology.orchestra.run.smithi002:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-17T00:06:32.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T00:06:32.489 DEBUG:teuthology.orchestra.run.smithi099:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-17T00:06:32.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T00:06:32.514 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-17T00:06:32.515 DEBUG:teuthology.orchestra.run.smithi002:> sudo /usr/sbin/setenforce permissive 2024-09-17T00:06:32.559 DEBUG:teuthology.orchestra.run.smithi099:> sudo /usr/sbin/setenforce permissive 2024-09-17T00:06:32.594 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-17T00:06:32.611 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-17T00:06:32.621 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-17T00:06:32.622 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi002.front.sepia.ceph.com,smithi099.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-17T00:10:20.940 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi002.front.sepia.ceph.com'), Remote(name='ubuntu@smithi099.front.sepia.ceph.com')] 2024-09-17T00:10:20.942 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi002.front.sepia.ceph.com' 2024-09-17T00:10:20.943 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi002.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T00:10:21.021 DEBUG:teuthology.orchestra.run.smithi002:> true 2024-09-17T00:10:21.097 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi002.front.sepia.ceph.com' 2024-09-17T00:10:21.097 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi099.front.sepia.ceph.com' 2024-09-17T00:10:21.098 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi099.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-17T00:10:21.174 DEBUG:teuthology.orchestra.run.smithi099:> true 2024-09-17T00:10:21.257 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi099.front.sepia.ceph.com' 2024-09-17T00:10:21.257 INFO:teuthology.run_tasks:Running task clock... 2024-09-17T00:10:21.267 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-17T00:10:21.267 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-17T00:10:21.268 DEBUG:teuthology.orchestra.run.smithi002:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-17T00:10:21.270 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-17T00:10:21.270 DEBUG:teuthology.orchestra.run.smithi099:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-17T00:10:21.299 INFO:teuthology.orchestra.run.smithi002.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-17T00:10:21.315 INFO:teuthology.orchestra.run.smithi002.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-17T00:10:21.326 INFO:teuthology.orchestra.run.smithi099.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-17T00:10:21.341 INFO:teuthology.orchestra.run.smithi099.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-17T00:10:21.341 INFO:teuthology.orchestra.run.smithi002.stderr:sudo: ntpd: command not found 2024-09-17T00:10:21.353 INFO:teuthology.orchestra.run.smithi002.stdout:506 Cannot talk to daemon 2024-09-17T00:10:21.365 INFO:teuthology.orchestra.run.smithi099.stderr:sudo: ntpd: command not found 2024-09-17T00:10:21.368 INFO:teuthology.orchestra.run.smithi002.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-17T00:10:21.376 INFO:teuthology.orchestra.run.smithi099.stdout:506 Cannot talk to daemon 2024-09-17T00:10:21.383 INFO:teuthology.orchestra.run.smithi002.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-17T00:10:21.391 INFO:teuthology.orchestra.run.smithi099.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-17T00:10:21.404 INFO:teuthology.orchestra.run.smithi099.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-17T00:10:21.446 INFO:teuthology.orchestra.run.smithi002.stderr:bash: line 1: ntpq: command not found 2024-09-17T00:10:21.450 INFO:teuthology.orchestra.run.smithi002.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-17T00:10:21.450 INFO:teuthology.orchestra.run.smithi002.stdout:=============================================================================== 2024-09-17T00:10:21.450 INFO:teuthology.orchestra.run.smithi002.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T00:10:21.450 INFO:teuthology.orchestra.run.smithi002.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T00:10:21.450 INFO:teuthology.orchestra.run.smithi002.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T00:10:21.450 INFO:teuthology.orchestra.run.smithi002.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T00:10:21.456 INFO:teuthology.orchestra.run.smithi099.stderr:bash: line 1: ntpq: command not found 2024-09-17T00:10:21.460 INFO:teuthology.orchestra.run.smithi099.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-17T00:10:21.460 INFO:teuthology.orchestra.run.smithi099.stdout:=============================================================================== 2024-09-17T00:10:21.460 INFO:teuthology.orchestra.run.smithi099.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T00:10:21.461 INFO:teuthology.orchestra.run.smithi099.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T00:10:21.461 INFO:teuthology.orchestra.run.smithi099.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T00:10:21.461 INFO:teuthology.orchestra.run.smithi099.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T00:10:21.462 INFO:teuthology.run_tasks:Running task cephadm... 2024-09-17T00:10:21.553 INFO:tasks.cephadm:Config: {'cephadm_branch': 'v17.2.0', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'image': 'quay.io/ceph/ceph:v17.2.0', 'conf': {'global': {'mon election default strategy': 1}, 'mgr': {'debug mgr': 20, 'debug ms': 1, 'mgr/cephadm/use_agent': False}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'CEPHADM_STRAY_DAEMON', 'CEPHADM_FAILED_DAEMON', 'CEPHADM_AGENT_DOWN'], 'log-only-match': ['CEPHADM_'], 'sha1': 'c2ddf9721ef254645201cdbabd9c8f49c30a586f'} 2024-09-17T00:10:21.553 INFO:tasks.cephadm:Cluster image is quay.io/ceph/ceph:v17.2.0 2024-09-17T00:10:21.554 INFO:tasks.cephadm:Cluster fsid is 3aad27a2-7489-11ef-bceb-c7b262605968 2024-09-17T00:10:21.554 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-09-17T00:10:21.554 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.2', 'mon.c': '[v2:172.21.15.2:3301,v1:172.21.15.2:6790]', 'mon.b': '172.21.15.99'} 2024-09-17T00:10:21.554 INFO:tasks.cephadm:First mon is mon.a on smithi002 2024-09-17T00:10:21.554 INFO:tasks.cephadm:First mgr is y 2024-09-17T00:10:21.555 INFO:tasks.cephadm:Normalizing hostnames... 2024-09-17T00:10:21.555 DEBUG:teuthology.orchestra.run.smithi002:> sudo hostname $(hostname -s) 2024-09-17T00:10:21.582 DEBUG:teuthology.orchestra.run.smithi099:> sudo hostname $(hostname -s) 2024-09-17T00:10:21.609 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref v17.2.0)... 2024-09-17T00:10:21.610 DEBUG:teuthology.orchestra.run.smithi002:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v17.2.0/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-09-17T00:10:21.811 INFO:teuthology.orchestra.run.smithi002.stdout:-rw-r--r--. 1 ubuntu ubuntu 320521 Sep 17 00:10 /home/ubuntu/cephtest/cephadm 2024-09-17T00:10:21.812 DEBUG:teuthology.orchestra.run.smithi099:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v17.2.0/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-09-17T00:10:21.941 INFO:teuthology.orchestra.run.smithi099.stdout:-rw-r--r--. 1 ubuntu ubuntu 320521 Sep 17 00:10 /home/ubuntu/cephtest/cephadm 2024-09-17T00:10:21.941 DEBUG:teuthology.orchestra.run.smithi002:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-09-17T00:10:21.958 DEBUG:teuthology.orchestra.run.smithi099:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-09-17T00:10:21.994 INFO:tasks.cephadm:Pulling image quay.io/ceph/ceph:v17.2.0 on all hosts... 2024-09-17T00:10:21.994 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 pull 2024-09-17T00:10:22.000 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 pull 2024-09-17T00:10:23.705 INFO:teuthology.orchestra.run.smithi002.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-09-17T00:10:23.710 INFO:teuthology.orchestra.run.smithi099.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-09-17T00:10:35.373 INFO:teuthology.orchestra.run.smithi099.stdout:{ 2024-09-17T00:10:35.374 INFO:teuthology.orchestra.run.smithi099.stdout: "ceph_version": "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)", 2024-09-17T00:10:35.374 INFO:teuthology.orchestra.run.smithi099.stdout: "image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", 2024-09-17T00:10:35.374 INFO:teuthology.orchestra.run.smithi099.stdout: "repo_digests": [ 2024-09-17T00:10:35.374 INFO:teuthology.orchestra.run.smithi099.stdout: "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", 2024-09-17T00:10:35.374 INFO:teuthology.orchestra.run.smithi099.stdout: "quay.io/ceph/ceph@sha256:cb4d698cb769b6aba05bf6ef04f41a7fe694160140347576e13bd9348514b667" 2024-09-17T00:10:35.374 INFO:teuthology.orchestra.run.smithi099.stdout: ] 2024-09-17T00:10:35.374 INFO:teuthology.orchestra.run.smithi099.stdout:} 2024-09-17T00:10:36.004 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-09-17T00:10:36.004 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph_version": "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)", 2024-09-17T00:10:36.004 INFO:teuthology.orchestra.run.smithi002.stdout: "image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", 2024-09-17T00:10:36.004 INFO:teuthology.orchestra.run.smithi002.stdout: "repo_digests": [ 2024-09-17T00:10:36.004 INFO:teuthology.orchestra.run.smithi002.stdout: "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", 2024-09-17T00:10:36.005 INFO:teuthology.orchestra.run.smithi002.stdout: "quay.io/ceph/ceph@sha256:cb4d698cb769b6aba05bf6ef04f41a7fe694160140347576e13bd9348514b667" 2024-09-17T00:10:36.005 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2024-09-17T00:10:36.005 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-09-17T00:10:36.022 DEBUG:teuthology.orchestra.run.smithi002:> sudo mkdir -p /etc/ceph 2024-09-17T00:10:36.077 DEBUG:teuthology.orchestra.run.smithi099:> sudo mkdir -p /etc/ceph 2024-09-17T00:10:36.107 DEBUG:teuthology.orchestra.run.smithi002:> sudo chmod 777 /etc/ceph 2024-09-17T00:10:36.133 DEBUG:teuthology.orchestra.run.smithi099:> sudo chmod 777 /etc/ceph 2024-09-17T00:10:36.174 INFO:tasks.cephadm:Writing seed config... 2024-09-17T00:10:36.175 INFO:tasks.cephadm: override: [global] mon election default strategy = 1 2024-09-17T00:10:36.175 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-09-17T00:10:36.175 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-09-17T00:10:36.176 INFO:tasks.cephadm: override: [mgr] mgr/cephadm/use_agent = False 2024-09-17T00:10:36.176 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-09-17T00:10:36.176 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-09-17T00:10:36.176 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-09-17T00:10:36.176 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-09-17T00:10:36.176 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-09-17T00:10:36.176 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-09-17T00:10:36.177 DEBUG:teuthology.orchestra.run.smithi002:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-09-17T00:10:36.192 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 3aad27a2-7489-11ef-bceb-c7b262605968 mon election default strategy = 1 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 mgr/cephadm/use_agent = False [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true 2024-09-17T00:10:36.192 DEBUG:teuthology.orchestra.run.smithi002:mon.a> sudo journalctl -f -n 0 -u ceph-3aad27a2-7489-11ef-bceb-c7b262605968@mon.a.service 2024-09-17T00:10:36.234 DEBUG:teuthology.orchestra.run.smithi002:mgr.y> sudo journalctl -f -n 0 -u ceph-3aad27a2-7489-11ef-bceb-c7b262605968@mgr.y.service 2024-09-17T00:10:36.277 INFO:tasks.cephadm:Bootstrapping... 2024-09-17T00:10:36.277 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 -v bootstrap --fsid 3aad27a2-7489-11ef-bceb-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.2 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-09-17T00:10:36.474 INFO:teuthology.orchestra.run.smithi002.stderr:-------------------------------------------------------------------------------- 2024-09-17T00:10:36.474 INFO:teuthology.orchestra.run.smithi002.stderr:cephadm ['--image', 'quay.io/ceph/ceph:v17.2.0', '-v', 'bootstrap', '--fsid', '3aad27a2-7489-11ef-bceb-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.2', '--skip-admin-label'] 2024-09-17T00:10:36.496 INFO:teuthology.orchestra.run.smithi002.stderr:/bin/podman: 5.2.2 2024-09-17T00:10:36.499 INFO:teuthology.orchestra.run.smithi002.stderr:Verifying podman|docker is present... 2024-09-17T00:10:36.519 INFO:teuthology.orchestra.run.smithi002.stderr:/bin/podman: 5.2.2 2024-09-17T00:10:36.521 INFO:teuthology.orchestra.run.smithi002.stderr:Verifying lvm2 is present... 2024-09-17T00:10:36.522 INFO:teuthology.orchestra.run.smithi002.stderr:Verifying time synchronization is in place... 2024-09-17T00:10:36.529 INFO:teuthology.orchestra.run.smithi002.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-09-17T00:10:36.535 INFO:teuthology.orchestra.run.smithi002.stderr:systemctl: inactive 2024-09-17T00:10:36.542 INFO:teuthology.orchestra.run.smithi002.stderr:systemctl: enabled 2024-09-17T00:10:36.549 INFO:teuthology.orchestra.run.smithi002.stderr:systemctl: active 2024-09-17T00:10:36.549 INFO:teuthology.orchestra.run.smithi002.stderr:Unit chronyd.service is enabled and running 2024-09-17T00:10:36.549 INFO:teuthology.orchestra.run.smithi002.stderr:Repeating the final host check... 2024-09-17T00:10:36.576 INFO:teuthology.orchestra.run.smithi002.stderr:/bin/podman: 5.2.2 2024-09-17T00:10:36.578 INFO:teuthology.orchestra.run.smithi002.stderr:podman (/bin/podman) version 5.2.2 is present 2024-09-17T00:10:36.578 INFO:teuthology.orchestra.run.smithi002.stderr:systemctl is present 2024-09-17T00:10:36.578 INFO:teuthology.orchestra.run.smithi002.stderr:lvcreate is present 2024-09-17T00:10:36.584 INFO:teuthology.orchestra.run.smithi002.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-09-17T00:10:36.591 INFO:teuthology.orchestra.run.smithi002.stderr:systemctl: inactive 2024-09-17T00:10:36.598 INFO:teuthology.orchestra.run.smithi002.stderr:systemctl: enabled 2024-09-17T00:10:36.605 INFO:teuthology.orchestra.run.smithi002.stderr:systemctl: active 2024-09-17T00:10:36.605 INFO:teuthology.orchestra.run.smithi002.stderr:Unit chronyd.service is enabled and running 2024-09-17T00:10:36.606 INFO:teuthology.orchestra.run.smithi002.stderr:Host looks OK 2024-09-17T00:10:36.606 INFO:teuthology.orchestra.run.smithi002.stderr:Cluster fsid: 3aad27a2-7489-11ef-bceb-c7b262605968 2024-09-17T00:10:36.606 INFO:teuthology.orchestra.run.smithi002.stderr:Acquiring lock 140063330214528 on /run/cephadm/3aad27a2-7489-11ef-bceb-c7b262605968.lock 2024-09-17T00:10:36.606 INFO:teuthology.orchestra.run.smithi002.stderr:Lock 140063330214528 acquired on /run/cephadm/3aad27a2-7489-11ef-bceb-c7b262605968.lock 2024-09-17T00:10:36.606 INFO:teuthology.orchestra.run.smithi002.stderr:Verifying IP 172.21.15.2 port 3300 ... 2024-09-17T00:10:36.606 INFO:teuthology.orchestra.run.smithi002.stderr:Verifying IP 172.21.15.2 port 6789 ... 2024-09-17T00:10:36.607 INFO:teuthology.orchestra.run.smithi002.stderr:Base mon IP is 172.21.15.2, final addrv is [v2:172.21.15.2:3300,v1:172.21.15.2:6789] 2024-09-17T00:10:36.609 INFO:teuthology.orchestra.run.smithi002.stderr:/sbin/ip: default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.2 metric 100 2024-09-17T00:10:36.609 INFO:teuthology.orchestra.run.smithi002.stderr:/sbin/ip: 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.2 metric 100 2024-09-17T00:10:36.612 INFO:teuthology.orchestra.run.smithi002.stderr:/sbin/ip: ::1 dev lo proto kernel metric 256 pref medium 2024-09-17T00:10:36.612 INFO:teuthology.orchestra.run.smithi002.stderr:/sbin/ip: fe80::/64 dev ens1f0 proto kernel metric 256 pref medium 2024-09-17T00:10:36.612 INFO:teuthology.orchestra.run.smithi002.stderr:/sbin/ip: default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 1024 expires 1788sec hoplimit 64 pref medium 2024-09-17T00:10:36.614 INFO:teuthology.orchestra.run.smithi002.stderr:/sbin/ip: 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-09-17T00:10:36.615 INFO:teuthology.orchestra.run.smithi002.stderr:/sbin/ip: inet6 ::1/128 scope host 2024-09-17T00:10:36.615 INFO:teuthology.orchestra.run.smithi002.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-09-17T00:10:36.615 INFO:teuthology.orchestra.run.smithi002.stderr:/sbin/ip: 4: ens1f0: mtu 1500 state UP qlen 1000 2024-09-17T00:10:36.615 INFO:teuthology.orchestra.run.smithi002.stderr:/sbin/ip: inet6 fe80::ec4:7aff:febd:1920/64 scope link 2024-09-17T00:10:36.615 INFO:teuthology.orchestra.run.smithi002.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-09-17T00:10:36.615 INFO:teuthology.orchestra.run.smithi002.stderr:Mon IP `172.21.15.2` is in CIDR network `172.21.0.0/20` 2024-09-17T00:10:36.615 INFO:teuthology.orchestra.run.smithi002.stderr:- internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-09-17T00:10:36.616 INFO:teuthology.orchestra.run.smithi002.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-09-17T00:10:36.720 INFO:teuthology.orchestra.run.smithi002.stderr:/bin/podman: Trying to pull quay.io/ceph/ceph:v17.2.0... 2024-09-17T00:10:37.047 INFO:teuthology.orchestra.run.smithi002.stderr:/bin/podman: Getting image source signatures 2024-09-17T00:10:37.047 INFO:teuthology.orchestra.run.smithi002.stderr:/bin/podman: Copying blob sha256:33ca8fff7868c4dc0c11e09bca97c720eb9cfbab7221216754367dd8de70388a 2024-09-17T00:10:37.047 INFO:teuthology.orchestra.run.smithi002.stderr:/bin/podman: Copying blob sha256:89b4a75bc2d8500f15463747507c9623df43886c134463e7f0527e70900e7a7b 2024-09-17T00:10:37.047 INFO:teuthology.orchestra.run.smithi002.stderr:/bin/podman: Copying blob sha256:c32ab78b488d0c72f64eded765c0cf6b5bf2c75dab66cb62a9d367fa6ec42513 2024-09-17T00:10:37.047 INFO:teuthology.orchestra.run.smithi002.stderr:/bin/podman: Copying blob sha256:599d07cb321ff0a3c82224e1138fc685793fa69b93ed5780415751a5f7e4b8c2 2024-09-17T00:10:37.047 INFO:teuthology.orchestra.run.smithi002.stderr:/bin/podman: Copying blob sha256:a70843738bb77e1ab9c1f85969ebdfa55f178e746be081d1cb4f94011f69eb7c 2024-09-17T00:10:37.047 INFO:teuthology.orchestra.run.smithi002.stderr:/bin/podman: Copying config sha256:e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 2024-09-17T00:10:37.215 INFO:teuthology.orchestra.run.smithi002.stderr:/bin/podman: Writing manifest to image destination 2024-09-17T00:10:37.368 INFO:teuthology.orchestra.run.smithi002.stderr:/bin/podman: e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 2024-09-17T00:10:37.805 INFO:teuthology.orchestra.run.smithi002.stderr:ceph: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable) 2024-09-17T00:10:38.292 INFO:teuthology.orchestra.run.smithi002.stderr:Ceph version: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable) 2024-09-17T00:10:38.292 INFO:teuthology.orchestra.run.smithi002.stderr:Extracting ceph user uid/gid from container image... 2024-09-17T00:10:38.692 INFO:teuthology.orchestra.run.smithi002.stderr:stat: 167 167 2024-09-17T00:10:39.297 INFO:teuthology.orchestra.run.smithi002.stderr:Creating initial keys... 2024-09-17T00:10:39.697 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph-authtool: AQD/yOhmYTFSJRAATgzS2J/FKQpJpMDbsbdxYQ== 2024-09-17T00:10:40.661 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph-authtool: AQAAyehmhDc4IhAAIrJHIaTdootVWVC1vgHs4A== 2024-09-17T00:10:41.626 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph-authtool: AQAByehmdgYGIRAAX7RhxaZkikXXdHx/fERDRQ== 2024-09-17T00:10:42.109 INFO:teuthology.orchestra.run.smithi002.stderr:Creating initial monmap... 2024-09-17T00:10:42.507 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: monmap file /tmp/monmap 2024-09-17T00:10:42.507 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/monmaptool: setting min_mon_release = octopus 2024-09-17T00:10:42.507 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: set fsid to 3aad27a2-7489-11ef-bceb-c7b262605968 2024-09-17T00:10:42.507 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-09-17T00:10:43.030 INFO:teuthology.orchestra.run.smithi002.stderr:monmaptool for a [v2:172.21.15.2:3300,v1:172.21.15.2:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-09-17T00:10:43.030 INFO:teuthology.orchestra.run.smithi002.stderr:setting min_mon_release = octopus 2024-09-17T00:10:43.030 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/monmaptool: set fsid to 3aad27a2-7489-11ef-bceb-c7b262605968 2024-09-17T00:10:43.030 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-09-17T00:10:43.030 INFO:teuthology.orchestra.run.smithi002.stderr: 2024-09-17T00:10:43.031 INFO:teuthology.orchestra.run.smithi002.stderr:Creating mon... 2024-09-17T00:10:43.952 INFO:teuthology.orchestra.run.smithi002.stderr:create mon.a on 2024-09-17T00:10:44.180 INFO:teuthology.orchestra.run.smithi002.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-09-17T00:10:44.340 INFO:teuthology.orchestra.run.smithi002.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-3aad27a2-7489-11ef-bceb-c7b262605968.target → /etc/systemd/system/ceph-3aad27a2-7489-11ef-bceb-c7b262605968.target. 2024-09-17T00:10:44.341 INFO:teuthology.orchestra.run.smithi002.stderr:systemctl: Created symlink /etc/systemd/system/ceph.target.wants/ceph-3aad27a2-7489-11ef-bceb-c7b262605968.target → /etc/systemd/system/ceph-3aad27a2-7489-11ef-bceb-c7b262605968.target. 2024-09-17T00:10:44.949 INFO:teuthology.orchestra.run.smithi002.stderr:systemctl: Failed to reset failed state of unit ceph-3aad27a2-7489-11ef-bceb-c7b262605968@mon.a.service: Unit ceph-3aad27a2-7489-11ef-bceb-c7b262605968@mon.a.service not loaded. 2024-09-17T00:10:44.957 INFO:teuthology.orchestra.run.smithi002.stderr:systemctl: Created symlink /etc/systemd/system/ceph-3aad27a2-7489-11ef-bceb-c7b262605968.target.wants/ceph-3aad27a2-7489-11ef-bceb-c7b262605968@mon.a.service → /etc/systemd/system/ceph-3aad27a2-7489-11ef-bceb-c7b262605968@.service. 2024-09-17T00:10:45.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:45 smithi002 podman[23376]: 2024-09-17 00:10:45.38616884 +0000 UTC m=+0.017486706 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph:v17.2.0 2024-09-17T00:10:46.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:46 smithi002 podman[23376]: 2024-09-17 00:10:46.425800336 +0000 UTC m=+1.057118185 container create 3dae30577d264c6e7cf0cbdb24918c7f074934c8e6a08aa09f8d480e433f8494 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mon-a, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vendor=Red Hat, Inc., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, build-date=2022-05-03T08:36:31.336870, io.openshift.expose-services=, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_BRANCH=HEAD, GIT_CLEAN=True, distribution-scope=public, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.19.8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base centos centos-stream, vcs-type=git, RELEASE=HEAD, release=754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, name=centos-stream, CEPH_POINT_RELEASE=-17.2.0, architecture=x86_64, version=8, com.redhat.component=centos-stream-container, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, maintainer=Guillaume Abrioux , com.redhat.license_terms=https://centos.org/legal/licensing-policy/, ceph=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.display-name=CentOS Stream 8) 2024-09-17T00:10:48.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:48 smithi002 podman[23376]: 2024-09-17 00:10:48.394361163 +0000 UTC m=+3.025679022 container init 3dae30577d264c6e7cf0cbdb24918c7f074934c8e6a08aa09f8d480e433f8494 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mon-a, ceph=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-type=git, io.openshift.tags=base centos centos-stream, com.redhat.component=centos-stream-container, GIT_BRANCH=HEAD, architecture=x86_64, build-date=2022-05-03T08:36:31.336870, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, CEPH_POINT_RELEASE=-17.2.0, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vendor=Red Hat, Inc., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.buildah.version=1.19.8, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.expose-services=, version=8, distribution-scope=public, io.k8s.display-name=CentOS Stream 8, GIT_CLEAN=True, release=754, name=centos-stream) 2024-09-17T00:10:48.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:48 smithi002 podman[23376]: 2024-09-17 00:10:48.398744047 +0000 UTC m=+3.030061893 container start 3dae30577d264c6e7cf0cbdb24918c7f074934c8e6a08aa09f8d480e433f8494 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mon-a, ceph=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., distribution-scope=public, io.openshift.tags=base centos centos-stream, name=centos-stream, io.k8s.display-name=CentOS Stream 8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, version=8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_BRANCH=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.expose-services=, CEPH_POINT_RELEASE=-17.2.0, architecture=x86_64, GIT_CLEAN=True, maintainer=Guillaume Abrioux , build-date=2022-05-03T08:36:31.336870, vendor=Red Hat, Inc., release=754, vcs-type=git, com.redhat.component=centos-stream-container, io.buildah.version=1.19.8, GIT_REPO=https://github.com/ceph/ceph-container.git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, RELEASE=HEAD) 2024-09-17T00:10:48.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:48 smithi002 ceph-mon[23394]: mkfs 3aad27a2-7489-11ef-bceb-c7b262605968 2024-09-17T00:10:48.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:48 smithi002 ceph-mon[23394]: mon.a is new leader, mons a in quorum (ranks 0) 2024-09-17T00:10:49.381 INFO:teuthology.orchestra.run.smithi002.stderr:systemctl: disabled 2024-09-17T00:10:49.387 INFO:teuthology.orchestra.run.smithi002.stderr:systemctl: inactive 2024-09-17T00:10:49.387 INFO:teuthology.orchestra.run.smithi002.stderr:firewalld.service is not enabled 2024-09-17T00:10:49.388 INFO:teuthology.orchestra.run.smithi002.stderr:Not possible to enable service . firewalld.service is not available 2024-09-17T00:10:49.388 INFO:teuthology.orchestra.run.smithi002.stderr:Waiting for mon to start... 2024-09-17T00:10:49.388 INFO:teuthology.orchestra.run.smithi002.stderr:Waiting for mon... 2024-09-17T00:10:49.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:49 smithi002 bash[23376]: 3dae30577d264c6e7cf0cbdb24918c7f074934c8e6a08aa09f8d480e433f8494 2024-09-17T00:10:49.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:49 smithi002 systemd[1]: Started Ceph mon.a for 3aad27a2-7489-11ef-bceb-c7b262605968. 2024-09-17T00:10:49.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:49 smithi002 ceph-mon[23394]: mon.a is new leader, mons a in quorum (ranks 0) 2024-09-17T00:10:49.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:49 smithi002 ceph-mon[23394]: monmap e1: 1 mons at {a=[v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0]} 2024-09-17T00:10:49.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:49 smithi002 ceph-mon[23394]: fsmap 2024-09-17T00:10:49.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:49 smithi002 ceph-mon[23394]: osdmap e1: 0 total, 0 up, 0 in 2024-09-17T00:10:49.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:49 smithi002 ceph-mon[23394]: mgrmap e1: no daemons active 2024-09-17T00:10:52.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:52 smithi002 ceph-mon[23394]: from='client.? 172.21.15.2:0/3676050904' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-09-17T00:10:53.240 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: cluster: 2024-09-17T00:10:53.241 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: id: 3aad27a2-7489-11ef-bceb-c7b262605968 2024-09-17T00:10:53.241 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: health: HEALTH_OK 2024-09-17T00:10:53.241 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2024-09-17T00:10:53.241 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: services: 2024-09-17T00:10:53.241 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: mon: 1 daemons, quorum a (age 4s) 2024-09-17T00:10:53.241 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: mgr: no daemons active 2024-09-17T00:10:53.241 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: osd: 0 osds: 0 up, 0 in 2024-09-17T00:10:53.241 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2024-09-17T00:10:53.241 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: data: 2024-09-17T00:10:53.241 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: pools: 0 pools, 0 pgs 2024-09-17T00:10:53.241 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: objects: 0 objects, 0 B 2024-09-17T00:10:53.241 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: usage: 0 B used, 0 B / 0 B avail 2024-09-17T00:10:53.241 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: pgs: 2024-09-17T00:10:53.242 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2024-09-17T00:10:54.017 INFO:teuthology.orchestra.run.smithi002.stderr:mon is available 2024-09-17T00:10:54.017 INFO:teuthology.orchestra.run.smithi002.stderr:Assimilating anything we can from ceph.conf... 2024-09-17T00:10:54.551 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2024-09-17T00:10:54.551 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: [global] 2024-09-17T00:10:54.551 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: fsid = 3aad27a2-7489-11ef-bceb-c7b262605968 2024-09-17T00:10:54.551 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: mon_host = [v2:172.21.15.2:3300,v1:172.21.15.2:6789] 2024-09-17T00:10:54.551 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-09-17T00:10:54.551 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-09-17T00:10:54.551 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-09-17T00:10:54.551 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-09-17T00:10:54.551 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2024-09-17T00:10:54.551 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: [mgr] 2024-09-17T00:10:54.551 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: mgr/cephadm/use_agent = False 2024-09-17T00:10:54.552 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-09-17T00:10:54.552 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2024-09-17T00:10:54.552 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: [osd] 2024-09-17T00:10:54.552 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-09-17T00:10:54.552 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-09-17T00:10:54.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:54 smithi002 ceph-mon[23394]: from='client.? 172.21.15.2:0/4176920548' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-09-17T00:10:54.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:54 smithi002 ceph-mon[23394]: from='client.? 172.21.15.2:0/4176920548' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-09-17T00:10:55.073 INFO:teuthology.orchestra.run.smithi002.stderr:Generating new minimal ceph.conf... 2024-09-17T00:10:55.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:55 smithi002 ceph-mon[23394]: from='client.? 172.21.15.2:0/3017894893' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:10:56.113 INFO:teuthology.orchestra.run.smithi002.stderr:Restarting the monitor... 2024-09-17T00:10:56.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:56 smithi002 systemd[1]: Stopping Ceph mon.a for 3aad27a2-7489-11ef-bceb-c7b262605968... 2024-09-17T00:10:56.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:56 smithi002 bash[23614]: Error: no container with name or ID "ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mon.a" found: no such container 2024-09-17T00:10:56.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:56 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mon-a[23390]: 2024-09-17T00:10:56.274+0000 7f0fed5b4700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-17T00:10:56.480 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:56 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mon-a[23390]: 2024-09-17T00:10:56.274+0000 7f0fed5b4700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-09-17T00:10:56.763 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:56 smithi002 podman[23625]: 2024-09-17 00:10:56.510638497 +0000 UTC m=+0.337587764 container died 3dae30577d264c6e7cf0cbdb24918c7f074934c8e6a08aa09f8d480e433f8494 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mon-a, io.openshift.expose-services=, build-date=2022-05-03T08:36:31.336870, RELEASE=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.19.8, version=8, vcs-type=git, io.openshift.tags=base centos centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, name=centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, release=754, ceph=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vendor=Red Hat, Inc., GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-17.2.0, com.redhat.component=centos-stream-container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, maintainer=Guillaume Abrioux , com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.display-name=CentOS Stream 8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., architecture=x86_64, distribution-scope=public, GIT_CLEAN=True) 2024-09-17T00:10:57.223 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:56 smithi002 podman[23625]: 2024-09-17 00:10:56.765707027 +0000 UTC m=+0.592656287 container cleanup 3dae30577d264c6e7cf0cbdb24918c7f074934c8e6a08aa09f8d480e433f8494 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mon-a, CEPH_POINT_RELEASE=-17.2.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , vendor=Red Hat, Inc., architecture=x86_64, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_BRANCH=HEAD, RELEASE=HEAD, io.k8s.display-name=CentOS Stream 8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.tags=base centos centos-stream, release=754, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, distribution-scope=public, version=8, ceph=True, build-date=2022-05-03T08:36:31.336870, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, name=centos-stream, com.redhat.component=centos-stream-container, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.expose-services=, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.19.8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-type=git) 2024-09-17T00:10:57.223 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:56 smithi002 bash[23625]: ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mon-a 2024-09-17T00:10:57.223 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:56 smithi002 bash[23653]: Error: no container with name or ID "ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mon.a" found: no such container 2024-09-17T00:10:57.223 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:56 smithi002 podman[23638]: 2024-09-17 00:10:56.983194017 +0000 UTC m=+0.469642585 container remove 3dae30577d264c6e7cf0cbdb24918c7f074934c8e6a08aa09f8d480e433f8494 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mon-a, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.display-name=CentOS Stream 8, CEPH_POINT_RELEASE=-17.2.0, io.openshift.expose-services=, distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-type=git, ceph=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, RELEASE=HEAD, GIT_BRANCH=HEAD, release=754, version=8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vendor=Red Hat, Inc., architecture=x86_64, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base centos centos-stream, build-date=2022-05-03T08:36:31.336870, io.buildah.version=1.19.8, maintainer=Guillaume Abrioux , com.redhat.component=centos-stream-container, name=centos-stream) 2024-09-17T00:10:57.223 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:56 smithi002 systemd[1]: ceph-3aad27a2-7489-11ef-bceb-c7b262605968@mon.a.service: Deactivated successfully. 2024-09-17T00:10:57.223 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:56 smithi002 systemd[1]: Stopped Ceph mon.a for 3aad27a2-7489-11ef-bceb-c7b262605968. 2024-09-17T00:10:57.223 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 systemd[1]: Starting Ceph mon.a for 3aad27a2-7489-11ef-bceb-c7b262605968... 2024-09-17T00:10:57.223 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 podman[23718]: 2024-09-17 00:10:57.128597743 +0000 UTC m=+0.017462365 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph:v17.2.0 2024-09-17T00:10:57.223 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 podman[23718]: 2024-09-17 00:10:57.225596011 +0000 UTC m=+0.114460669 container create a68823df625f16cc31a5afdf232d2cb15ad7217d72d4553d92573f32439f1f15 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mon-a, io.openshift.tags=base centos centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_CLEAN=True, build-date=2022-05-03T08:36:31.336870, io.k8s.display-name=CentOS Stream 8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, name=centos-stream, RELEASE=HEAD, CEPH_POINT_RELEASE=-17.2.0, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, release=754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Guillaume Abrioux , io.openshift.expose-services=, vendor=Red Hat, Inc., GIT_BRANCH=HEAD, distribution-scope=public, io.buildah.version=1.19.8, com.redhat.component=centos-stream-container, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-type=git, version=8) 2024-09-17T00:10:57.474 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 podman[23718]: 2024-09-17 00:10:57.453487305 +0000 UTC m=+0.342351928 container init a68823df625f16cc31a5afdf232d2cb15ad7217d72d4553d92573f32439f1f15 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mon-a, GIT_BRANCH=HEAD, io.k8s.display-name=CentOS Stream 8, version=8, build-date=2022-05-03T08:36:31.336870, vcs-type=git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.expose-services=, maintainer=Guillaume Abrioux , name=centos-stream, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, RELEASE=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, CEPH_POINT_RELEASE=-17.2.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.component=centos-stream-container, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.tags=base centos centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, distribution-scope=public, ceph=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.buildah.version=1.19.8, GIT_CLEAN=True, vendor=Red Hat, Inc., GIT_REPO=https://github.com/ceph/ceph-container.git, release=754) 2024-09-17T00:10:57.475 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 podman[23718]: 2024-09-17 00:10:57.457016331 +0000 UTC m=+0.345880946 container start a68823df625f16cc31a5afdf232d2cb15ad7217d72d4553d92573f32439f1f15 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mon-a, build-date=2022-05-03T08:36:31.336870, vcs-type=git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.display-name=CentOS Stream 8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., RELEASE=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, release=754, version=8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, ceph=True, distribution-scope=public, io.openshift.expose-services=, architecture=x86_64, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.component=centos-stream-container, maintainer=Guillaume Abrioux , io.buildah.version=1.19.8, CEPH_POINT_RELEASE=-17.2.0, vendor=Red Hat, Inc., name=centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_CLEAN=True, io.openshift.tags=base centos centos-stream) 2024-09-17T00:10:57.536 INFO:teuthology.orchestra.run.smithi002.stderr:Setting mon public_network to 172.21.0.0/20 2024-09-17T00:10:57.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: set uid:gid to 167:167 (ceph:ceph) 2024-09-17T00:10:57.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable), process ceph-mon, pid 2 2024-09-17T00:10:57.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: pidfile_write: ignore empty --pid-file 2024-09-17T00:10:57.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: load: jerasure load: lrc 2024-09-17T00:10:57.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: RocksDB version: 6.15.5 2024-09-17T00:10:57.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-09-17T00:10:57.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Compile date Apr 18 2022 2024-09-17T00:10:57.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: DB SUMMARY 2024-09-17T00:10:57.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: DB Session ID: XNXVE7ECQMG7OX1KCIK5 2024-09-17T00:10:57.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: CURRENT file: CURRENT 2024-09-17T00:10:57.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: IDENTITY file: IDENTITY 2024-09-17T00:10:57.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: MANIFEST file: MANIFEST-000009 size: 131 Bytes 2024-09-17T00:10:57.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-09-17T00:10:57.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000010.log size: 83405 ; 2024-09-17T00:10:57.731 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.error_if_exists: 0 2024-09-17T00:10:57.731 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.create_if_missing: 0 2024-09-17T00:10:57.731 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.paranoid_checks: 1 2024-09-17T00:10:57.731 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-17T00:10:57.731 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.env: 0x5603056c6860 2024-09-17T00:10:57.731 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.fs: Posix File System 2024-09-17T00:10:57.731 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.info_log: 0x5603078a5dc0 2024-09-17T00:10:57.731 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.max_file_opening_threads: 16 2024-09-17T00:10:57.731 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.statistics: (nil) 2024-09-17T00:10:57.731 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.use_fsync: 0 2024-09-17T00:10:57.731 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.max_log_file_size: 0 2024-09-17T00:10:57.731 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-17T00:10:57.732 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-17T00:10:57.732 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.keep_log_file_num: 1000 2024-09-17T00:10:57.732 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.recycle_log_file_num: 0 2024-09-17T00:10:57.732 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.allow_fallocate: 1 2024-09-17T00:10:57.732 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.allow_mmap_reads: 0 2024-09-17T00:10:57.732 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.allow_mmap_writes: 0 2024-09-17T00:10:57.732 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.use_direct_reads: 0 2024-09-17T00:10:57.732 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-17T00:10:57.732 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.create_missing_column_families: 0 2024-09-17T00:10:57.732 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.db_log_dir: 2024-09-17T00:10:57.732 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-a/store.db 2024-09-17T00:10:57.732 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-17T00:10:57.732 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-17T00:10:57.732 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-17T00:10:57.733 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-17T00:10:57.733 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-17T00:10:57.733 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-17T00:10:57.733 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.advise_random_on_open: 1 2024-09-17T00:10:57.733 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.db_write_buffer_size: 0 2024-09-17T00:10:57.733 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.write_buffer_manager: 0x5603079962a0 2024-09-17T00:10:57.733 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-17T00:10:57.733 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-09-17T00:10:57.733 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-17T00:10:57.733 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-17T00:10:57.733 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.rate_limiter: (nil) 2024-09-17T00:10:57.733 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-17T00:10:57.733 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.wal_recovery_mode: 2 2024-09-17T00:10:57.733 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.enable_thread_tracking: 0 2024-09-17T00:10:57.734 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.enable_pipelined_write: 0 2024-09-17T00:10:57.734 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.unordered_write: 0 2024-09-17T00:10:57.734 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-17T00:10:57.734 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-17T00:10:57.734 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-17T00:10:57.734 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-17T00:10:57.734 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.row_cache: None 2024-09-17T00:10:57.734 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.wal_filter: None 2024-09-17T00:10:57.734 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-17T00:10:57.734 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.allow_ingest_behind: 0 2024-09-17T00:10:57.734 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.preserve_deletes: 0 2024-09-17T00:10:57.734 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.two_write_queues: 0 2024-09-17T00:10:57.734 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.manual_wal_flush: 0 2024-09-17T00:10:57.735 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.atomic_flush: 0 2024-09-17T00:10:57.735 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-17T00:10:57.735 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-17T00:10:57.735 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-17T00:10:57.735 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.log_readahead_size: 0 2024-09-17T00:10:57.735 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-17T00:10:57.735 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.best_efforts_recovery: 0 2024-09-17T00:10:57.735 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-17T00:10:57.735 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-17T00:10:57.735 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.allow_data_in_errors: 0 2024-09-17T00:10:57.735 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.db_host_id: __hostname__ 2024-09-17T00:10:57.736 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.max_background_jobs: 2 2024-09-17T00:10:57.738 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.max_background_compactions: -1 2024-09-17T00:10:57.738 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.max_subcompactions: 1 2024-09-17T00:10:57.738 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-17T00:10:57.738 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-17T00:10:57.738 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-17T00:10:57.738 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.max_total_wal_size: 0 2024-09-17T00:10:57.738 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-17T00:10:57.738 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-17T00:10:57.739 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-17T00:10:57.739 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-17T00:10:57.739 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.max_open_files: -1 2024-09-17T00:10:57.739 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.bytes_per_sync: 0 2024-09-17T00:10:57.739 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-17T00:10:57.739 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-17T00:10:57.739 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.compaction_readahead_size: 0 2024-09-17T00:10:57.739 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.max_background_flushes: -1 2024-09-17T00:10:57.739 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Compression algorithms supported: 2024-09-17T00:10:57.739 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-17T00:10:57.739 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: kZSTD supported: 0 2024-09-17T00:10:57.740 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: kXpressCompression supported: 0 2024-09-17T00:10:57.740 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: kLZ4HCCompression supported: 1 2024-09-17T00:10:57.740 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: kLZ4Compression supported: 1 2024-09-17T00:10:57.740 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: kBZip2Compression supported: 0 2024-09-17T00:10:57.740 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: kZlibCompression supported: 1 2024-09-17T00:10:57.740 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: kSnappyCompression supported: 1 2024-09-17T00:10:57.740 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-17T00:10:57.740 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: [db/version_set.cc:4725] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000009 2024-09-17T00:10:57.740 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: [db/column_family.cc:597] --------------- Options for column family [default]: 2024-09-17T00:10:57.740 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-17T00:10:57.740 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.merge_operator: 2024-09-17T00:10:57.740 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.compaction_filter: None 2024-09-17T00:10:57.740 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.compaction_filter_factory: None 2024-09-17T00:10:57.741 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.sst_partitioner_factory: None 2024-09-17T00:10:57.741 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-17T00:10:57.741 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-17T00:10:57.741 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x560307871d00) 2024-09-17T00:10:57.741 INFO:journalctl@ceph.mon.a.smithi002.stdout: cache_index_and_filter_blocks: 1 2024-09-17T00:10:57.741 INFO:journalctl@ceph.mon.a.smithi002.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-17T00:10:57.741 INFO:journalctl@ceph.mon.a.smithi002.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-17T00:10:57.741 INFO:journalctl@ceph.mon.a.smithi002.stdout: pin_top_level_index_and_filter: 1 2024-09-17T00:10:57.741 INFO:journalctl@ceph.mon.a.smithi002.stdout: index_type: 0 2024-09-17T00:10:57.741 INFO:journalctl@ceph.mon.a.smithi002.stdout: data_block_index_type: 0 2024-09-17T00:10:57.741 INFO:journalctl@ceph.mon.a.smithi002.stdout: index_shortening: 1 2024-09-17T00:10:57.741 INFO:journalctl@ceph.mon.a.smithi002.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-17T00:10:57.741 INFO:journalctl@ceph.mon.a.smithi002.stdout: hash_index_allow_collision: 1 2024-09-17T00:10:57.742 INFO:journalctl@ceph.mon.a.smithi002.stdout: checksum: 1 2024-09-17T00:10:57.742 INFO:journalctl@ceph.mon.a.smithi002.stdout: no_block_cache: 0 2024-09-17T00:10:57.742 INFO:journalctl@ceph.mon.a.smithi002.stdout: block_cache: 0x5603078dc170 2024-09-17T00:10:57.742 INFO:journalctl@ceph.mon.a.smithi002.stdout: block_cache_name: BinnedLRUCache 2024-09-17T00:10:57.742 INFO:journalctl@ceph.mon.a.smithi002.stdout: block_cache_options: 2024-09-17T00:10:57.742 INFO:journalctl@ceph.mon.a.smithi002.stdout: capacity : 536870912 2024-09-17T00:10:57.742 INFO:journalctl@ceph.mon.a.smithi002.stdout: num_shard_bits : 4 2024-09-17T00:10:57.742 INFO:journalctl@ceph.mon.a.smithi002.stdout: strict_capacity_limit : 0 2024-09-17T00:10:57.742 INFO:journalctl@ceph.mon.a.smithi002.stdout: high_pri_pool_ratio: 0.000 2024-09-17T00:10:57.743 INFO:journalctl@ceph.mon.a.smithi002.stdout: block_cache_compressed: (nil) 2024-09-17T00:10:57.743 INFO:journalctl@ceph.mon.a.smithi002.stdout: persistent_cache: (nil) 2024-09-17T00:10:57.743 INFO:journalctl@ceph.mon.a.smithi002.stdout: block_size: 4096 2024-09-17T00:10:57.743 INFO:journalctl@ceph.mon.a.smithi002.stdout: block_size_deviation: 10 2024-09-17T00:10:57.743 INFO:journalctl@ceph.mon.a.smithi002.stdout: block_restart_interval: 16 2024-09-17T00:10:57.743 INFO:journalctl@ceph.mon.a.smithi002.stdout: index_block_restart_interval: 1 2024-09-17T00:10:57.743 INFO:journalctl@ceph.mon.a.smithi002.stdout: metadata_block_size: 4096 2024-09-17T00:10:57.743 INFO:journalctl@ceph.mon.a.smithi002.stdout: partition_filters: 0 2024-09-17T00:10:57.743 INFO:journalctl@ceph.mon.a.smithi002.stdout: use_delta_encoding: 1 2024-09-17T00:10:57.743 INFO:journalctl@ceph.mon.a.smithi002.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-09-17T00:10:57.743 INFO:journalctl@ceph.mon.a.smithi002.stdout: whole_key_filtering: 1 2024-09-17T00:10:57.743 INFO:journalctl@ceph.mon.a.smithi002.stdout: verify_compression: 0 2024-09-17T00:10:57.743 INFO:journalctl@ceph.mon.a.smithi002.stdout: read_amp_bytes_per_bit: 0 2024-09-17T00:10:57.743 INFO:journalctl@ceph.mon.a.smithi002.stdout: format_version: 4 2024-09-17T00:10:57.743 INFO:journalctl@ceph.mon.a.smithi002.stdout: enable_index_compression: 1 2024-09-17T00:10:57.744 INFO:journalctl@ceph.mon.a.smithi002.stdout: block_align: 0 2024-09-17T00:10:57.744 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.write_buffer_size: 33554432 2024-09-17T00:10:57.744 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.max_write_buffer_number: 2 2024-09-17T00:10:57.744 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.compression: NoCompression 2024-09-17T00:10:57.744 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.bottommost_compression: Disabled 2024-09-17T00:10:57.744 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.prefix_extractor: nullptr 2024-09-17T00:10:57.744 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-17T00:10:57.744 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.num_levels: 7 2024-09-17T00:10:57.744 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-17T00:10:57.744 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-17T00:10:57.744 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-17T00:10:57.744 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-17T00:10:57.744 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-17T00:10:57.744 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-17T00:10:57.745 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-17T00:10:57.745 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-17T00:10:57.745 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-17T00:10:57.745 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-17T00:10:57.745 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-17T00:10:57.745 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.compression_opts.level: 32767 2024-09-17T00:10:57.745 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.compression_opts.strategy: 0 2024-09-17T00:10:57.745 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-17T00:10:57.745 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-17T00:10:57.745 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-17T00:10:57.745 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.compression_opts.enabled: false 2024-09-17T00:10:57.745 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-17T00:10:57.745 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-17T00:10:57.745 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-17T00:10:57.746 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.target_file_size_base: 67108864 2024-09-17T00:10:57.746 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-17T00:10:57.746 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-17T00:10:57.746 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-17T00:10:57.746 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-17T00:10:57.746 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-17T00:10:57.746 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-17T00:10:57.746 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-17T00:10:57.746 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-17T00:10:57.746 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-17T00:10:57.746 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-17T00:10:57.746 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-17T00:10:57.746 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-17T00:10:57.746 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-17T00:10:57.746 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.arena_block_size: 4194304 2024-09-17T00:10:57.747 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-17T00:10:57.747 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-17T00:10:57.747 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-09-17T00:10:57.747 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.disable_auto_compactions: 0 2024-09-17T00:10:57.747 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-17T00:10:57.747 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-17T00:10:57.747 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-17T00:10:57.747 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-17T00:10:57.747 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-17T00:10:57.747 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-17T00:10:57.747 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-17T00:10:57.747 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-17T00:10:57.747 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-17T00:10:57.747 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-17T00:10:57.747 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.table_properties_collectors: 2024-09-17T00:10:57.748 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.inplace_update_support: 0 2024-09-17T00:10:57.748 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-17T00:10:57.748 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-17T00:10:57.748 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-17T00:10:57.748 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-17T00:10:57.748 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.bloom_locality: 0 2024-09-17T00:10:57.748 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.max_successive_merges: 0 2024-09-17T00:10:57.748 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-17T00:10:57.748 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.paranoid_file_checks: 0 2024-09-17T00:10:57.748 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.force_consistency_checks: 1 2024-09-17T00:10:57.748 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.report_bg_io_stats: 0 2024-09-17T00:10:57.748 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.ttl: 2592000 2024-09-17T00:10:57.748 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-17T00:10:57.748 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.enable_blob_files: false 2024-09-17T00:10:57.749 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.min_blob_size: 0 2024-09-17T00:10:57.749 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.blob_file_size: 268435456 2024-09-17T00:10:57.749 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-17T00:10:57.749 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-17T00:10:57.749 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-17T00:10:57.749 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: [db/version_set.cc:4773] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 11, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-09-17T00:10:57.749 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: [db/version_set.cc:4782] Column family [default] (ID 0), log number is 5 2024-09-17T00:10:57.749 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: [db/version_set.cc:4083] Creating manifest 13 2024-09-17T00:10:57.749 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726531857484342, "job": 1, "event": "recovery_started", "wal_files": [10]} 2024-09-17T00:10:57.749 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: [db/db_impl/db_impl_open.cc:847] Recovering log #10 mode 2 2024-09-17T00:10:57.749 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: [table/block_based/filter_policy.cc:996] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-09-17T00:10:57.749 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726531857485118, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 14, "file_size": 79601, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 77845, "index_size": 244, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 581, "raw_key_size": 10073, "raw_average_key_size": 47, "raw_value_size": 72096, "raw_average_value_size": 336, "num_data_blocks": 11, "num_entries": 214, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1726531857, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "45cae19e-03ac-46b2-a1b3-52301bc67af3", "db_session_id": "XNXVE7ECQMG7OX1KCIK5"}} 2024-09-17T00:10:57.749 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: [db/version_set.cc:4083] Creating manifest 15 2024-09-17T00:10:57.749 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726531857485465, "job": 1, "event": "recovery_finished"} 2024-09-17T00:10:57.750 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: [file/delete_scheduler.cc:73] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000010.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-17T00:10:57.750 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: [db/db_impl/db_impl_open.cc:1701] SstFileManager instance 0x5603078c2a80 2024-09-17T00:10:57.750 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: DB pointer 0x5603078d2000 2024-09-17T00:10:57.750 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: [db/db_impl/db_impl.cc:902] ------- DUMPING STATS ------- 2024-09-17T00:10:57.750 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-09-17T00:10:57.750 INFO:journalctl@ceph.mon.a.smithi002.stdout: ** DB Stats ** 2024-09-17T00:10:57.750 INFO:journalctl@ceph.mon.a.smithi002.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-17T00:10:57.751 INFO:journalctl@ceph.mon.a.smithi002.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-17T00:10:57.751 INFO:journalctl@ceph.mon.a.smithi002.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-17T00:10:57.751 INFO:journalctl@ceph.mon.a.smithi002.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-17T00:10:57.751 INFO:journalctl@ceph.mon.a.smithi002.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-17T00:10:57.751 INFO:journalctl@ceph.mon.a.smithi002.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-09-17T00:10:57.751 INFO:journalctl@ceph.mon.a.smithi002.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-17T00:10:57.751 INFO:journalctl@ceph.mon.a.smithi002.stdout: 2024-09-17T00:10:57.751 INFO:journalctl@ceph.mon.a.smithi002.stdout: ** Compaction Stats [default] ** 2024-09-17T00:10:57.751 INFO:journalctl@ceph.mon.a.smithi002.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-17T00:10:57.751 INFO:journalctl@ceph.mon.a.smithi002.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-17T00:10:57.751 INFO:journalctl@ceph.mon.a.smithi002.stdout: L0 2/0 79.50 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 151.5 0.00 0.00 1 0.001 0 0 2024-09-17T00:10:57.751 INFO:journalctl@ceph.mon.a.smithi002.stdout: Sum 2/0 79.50 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 151.5 0.00 0.00 1 0.001 0 0 2024-09-17T00:10:57.751 INFO:journalctl@ceph.mon.a.smithi002.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 151.5 0.00 0.00 1 0.001 0 0 2024-09-17T00:10:57.752 INFO:journalctl@ceph.mon.a.smithi002.stdout: 2024-09-17T00:10:57.752 INFO:journalctl@ceph.mon.a.smithi002.stdout: ** Compaction Stats [default] ** 2024-09-17T00:10:57.752 INFO:journalctl@ceph.mon.a.smithi002.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-17T00:10:57.752 INFO:journalctl@ceph.mon.a.smithi002.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-17T00:10:57.752 INFO:journalctl@ceph.mon.a.smithi002.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 151.5 0.00 0.00 1 0.001 0 0 2024-09-17T00:10:57.752 INFO:journalctl@ceph.mon.a.smithi002.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-17T00:10:57.752 INFO:journalctl@ceph.mon.a.smithi002.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-17T00:10:57.752 INFO:journalctl@ceph.mon.a.smithi002.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-17T00:10:57.752 INFO:journalctl@ceph.mon.a.smithi002.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-17T00:10:57.752 INFO:journalctl@ceph.mon.a.smithi002.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-17T00:10:57.752 INFO:journalctl@ceph.mon.a.smithi002.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-17T00:10:57.752 INFO:journalctl@ceph.mon.a.smithi002.stdout: Cumulative compaction: 0.00 GB write, 23.55 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-17T00:10:57.752 INFO:journalctl@ceph.mon.a.smithi002.stdout: Interval compaction: 0.00 GB write, 23.55 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-17T00:10:57.752 INFO:journalctl@ceph.mon.a.smithi002.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-17T00:10:57.753 INFO:journalctl@ceph.mon.a.smithi002.stdout: 2024-09-17T00:10:57.753 INFO:journalctl@ceph.mon.a.smithi002.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-17T00:10:57.753 INFO:journalctl@ceph.mon.a.smithi002.stdout: 2024-09-17T00:10:57.753 INFO:journalctl@ceph.mon.a.smithi002.stdout: ** Compaction Stats [default] ** 2024-09-17T00:10:57.753 INFO:journalctl@ceph.mon.a.smithi002.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-17T00:10:57.753 INFO:journalctl@ceph.mon.a.smithi002.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-17T00:10:57.753 INFO:journalctl@ceph.mon.a.smithi002.stdout: L0 2/0 79.50 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 151.5 0.00 0.00 1 0.001 0 0 2024-09-17T00:10:57.753 INFO:journalctl@ceph.mon.a.smithi002.stdout: Sum 2/0 79.50 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 151.5 0.00 0.00 1 0.001 0 0 2024-09-17T00:10:57.753 INFO:journalctl@ceph.mon.a.smithi002.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-09-17T00:10:57.753 INFO:journalctl@ceph.mon.a.smithi002.stdout: 2024-09-17T00:10:57.753 INFO:journalctl@ceph.mon.a.smithi002.stdout: ** Compaction Stats [default] ** 2024-09-17T00:10:57.753 INFO:journalctl@ceph.mon.a.smithi002.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-17T00:10:57.753 INFO:journalctl@ceph.mon.a.smithi002.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-17T00:10:57.753 INFO:journalctl@ceph.mon.a.smithi002.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 151.5 0.00 0.00 1 0.001 0 0 2024-09-17T00:10:57.753 INFO:journalctl@ceph.mon.a.smithi002.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-17T00:10:57.754 INFO:journalctl@ceph.mon.a.smithi002.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-17T00:10:57.754 INFO:journalctl@ceph.mon.a.smithi002.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-17T00:10:57.754 INFO:journalctl@ceph.mon.a.smithi002.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-17T00:10:57.754 INFO:journalctl@ceph.mon.a.smithi002.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-17T00:10:57.754 INFO:journalctl@ceph.mon.a.smithi002.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-17T00:10:57.754 INFO:journalctl@ceph.mon.a.smithi002.stdout: Cumulative compaction: 0.00 GB write, 23.33 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-17T00:10:57.754 INFO:journalctl@ceph.mon.a.smithi002.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-17T00:10:57.754 INFO:journalctl@ceph.mon.a.smithi002.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-17T00:10:57.754 INFO:journalctl@ceph.mon.a.smithi002.stdout: 2024-09-17T00:10:57.754 INFO:journalctl@ceph.mon.a.smithi002.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-17T00:10:57.754 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: starting mon.a rank 0 at public addrs [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] at bind addrs [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 3aad27a2-7489-11ef-bceb-c7b262605968 2024-09-17T00:10:57.754 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: mon.a@-1(???) e1 preinit fsid 3aad27a2-7489-11ef-bceb-c7b262605968 2024-09-17T00:10:57.754 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: mon.a@-1(???).mds e1 new map 2024-09-17T00:10:57.755 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: mon.a@-1(???).mds e1 print_map 2024-09-17T00:10:57.755 INFO:journalctl@ceph.mon.a.smithi002.stdout: e1 2024-09-17T00:10:57.755 INFO:journalctl@ceph.mon.a.smithi002.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-17T00:10:57.755 INFO:journalctl@ceph.mon.a.smithi002.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-09-17T00:10:57.755 INFO:journalctl@ceph.mon.a.smithi002.stdout: legacy client fscid: -1 2024-09-17T00:10:57.755 INFO:journalctl@ceph.mon.a.smithi002.stdout: 2024-09-17T00:10:57.755 INFO:journalctl@ceph.mon.a.smithi002.stdout: No filesystems configured 2024-09-17T00:10:57.755 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-09-17T00:10:57.755 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-17T00:10:57.755 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-17T00:10:57.755 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-17T00:10:57.755 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-09-17T00:10:57.755 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 bash[23718]: a68823df625f16cc31a5afdf232d2cb15ad7217d72d4553d92573f32439f1f15 2024-09-17T00:10:57.755 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 systemd[1]: Started Ceph mon.a for 3aad27a2-7489-11ef-bceb-c7b262605968. 2024-09-17T00:10:57.756 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: mon.a is new leader, mons a in quorum (ranks 0) 2024-09-17T00:10:57.756 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: monmap e1: 1 mons at {a=[v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0]} 2024-09-17T00:10:57.756 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: fsmap 2024-09-17T00:10:57.756 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: osdmap e1: 0 total, 0 up, 0 in 2024-09-17T00:10:57.756 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:57 smithi002 ceph-mon[23737]: mgrmap e1: no daemons active 2024-09-17T00:10:58.560 INFO:teuthology.orchestra.run.smithi002.stderr:Wrote config to /etc/ceph/ceph.conf 2024-09-17T00:10:58.560 INFO:teuthology.orchestra.run.smithi002.stderr:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-09-17T00:10:58.560 INFO:teuthology.orchestra.run.smithi002.stderr:Creating mgr... 2024-09-17T00:10:58.560 INFO:teuthology.orchestra.run.smithi002.stderr:Verifying port 9283 ... 2024-09-17T00:10:58.802 INFO:teuthology.orchestra.run.smithi002.stderr:systemctl: Failed to reset failed state of unit ceph-3aad27a2-7489-11ef-bceb-c7b262605968@mgr.y.service: Unit ceph-3aad27a2-7489-11ef-bceb-c7b262605968@mgr.y.service not loaded. 2024-09-17T00:10:58.810 INFO:teuthology.orchestra.run.smithi002.stderr:systemctl: Created symlink /etc/systemd/system/ceph-3aad27a2-7489-11ef-bceb-c7b262605968.target.wants/ceph-3aad27a2-7489-11ef-bceb-c7b262605968@mgr.y.service → /etc/systemd/system/ceph-3aad27a2-7489-11ef-bceb-c7b262605968@.service. 2024-09-17T00:10:59.390 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:10:59 smithi002 ceph-mon[23737]: from='client.? 172.21.15.2:0/146787078' entity='client.admin' 2024-09-17T00:10:59.390 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:10:59 smithi002 systemd[1]: Starting Ceph mgr.y for 3aad27a2-7489-11ef-bceb-c7b262605968... 2024-09-17T00:10:59.729 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:10:59 smithi002 podman[23949]: 2024-09-17 00:10:59.296448758 +0000 UTC m=+0.016716187 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph:v17.2.0 2024-09-17T00:11:00.729 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:11:00 smithi002 podman[23949]: 2024-09-17 00:11:00.306151626 +0000 UTC m=+1.026419057 container create d32b04d294d30eed382adf7e696bae32612feb16e79fe2ba590c23be6d622ab7 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_CLEAN=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.tags=base centos centos-stream, vendor=Red Hat, Inc., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vcs-type=git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, ceph=True, build-date=2022-05-03T08:36:31.336870, CEPH_POINT_RELEASE=-17.2.0, version=8, name=centos-stream, maintainer=Guillaume Abrioux , RELEASE=HEAD, architecture=x86_64, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_BRANCH=HEAD, io.k8s.display-name=CentOS Stream 8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, release=754, com.redhat.component=centos-stream-container, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.19.8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, distribution-scope=public) 2024-09-17T00:11:01.969 INFO:teuthology.orchestra.run.smithi002.stderr:systemctl: disabled 2024-09-17T00:11:01.977 INFO:teuthology.orchestra.run.smithi002.stderr:systemctl: inactive 2024-09-17T00:11:01.978 INFO:teuthology.orchestra.run.smithi002.stderr:firewalld.service is not enabled 2024-09-17T00:11:01.978 INFO:teuthology.orchestra.run.smithi002.stderr:Not possible to enable service . firewalld.service is not available 2024-09-17T00:11:01.989 INFO:teuthology.orchestra.run.smithi002.stderr:systemctl: disabled 2024-09-17T00:11:01.998 INFO:teuthology.orchestra.run.smithi002.stderr:systemctl: inactive 2024-09-17T00:11:01.999 INFO:teuthology.orchestra.run.smithi002.stderr:firewalld.service is not enabled 2024-09-17T00:11:01.999 INFO:teuthology.orchestra.run.smithi002.stderr:Not possible to open ports <[9283]>. firewalld.service is not available 2024-09-17T00:11:01.999 INFO:teuthology.orchestra.run.smithi002.stderr:Waiting for mgr to start... 2024-09-17T00:11:01.999 INFO:teuthology.orchestra.run.smithi002.stderr:Waiting for mgr... 2024-09-17T00:11:02.133 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:11:01 smithi002 podman[23949]: 2024-09-17 00:11:01.883177145 +0000 UTC m=+2.603444573 container init d32b04d294d30eed382adf7e696bae32612feb16e79fe2ba590c23be6d622ab7 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y, io.openshift.tags=base centos centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, release=754, GIT_BRANCH=HEAD, distribution-scope=public, ceph=True, build-date=2022-05-03T08:36:31.336870, version=8, CEPH_POINT_RELEASE=-17.2.0, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.component=centos-stream-container, io.openshift.expose-services=, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, maintainer=Guillaume Abrioux , summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., GIT_CLEAN=True, architecture=x86_64, vcs-type=git, io.buildah.version=1.19.8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, name=centos-stream, RELEASE=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.display-name=CentOS Stream 8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com) 2024-09-17T00:11:02.133 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:11:01 smithi002 podman[23949]: 2024-09-17 00:11:01.887165918 +0000 UTC m=+2.607433347 container start d32b04d294d30eed382adf7e696bae32612feb16e79fe2ba590c23be6d622ab7 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y, version=8, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux , name=centos-stream, architecture=x86_64, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-05-03T08:36:31.336870, GIT_BRANCH=HEAD, RELEASE=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, ceph=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=centos-stream-container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vendor=Red Hat, Inc., distribution-scope=public, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.buildah.version=1.19.8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.tags=base centos centos-stream, io.openshift.expose-services=, CEPH_POINT_RELEASE=-17.2.0, GIT_CLEAN=True, vcs-type=git, io.k8s.display-name=CentOS Stream 8, release=754) 2024-09-17T00:11:02.133 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:11:01 smithi002 bash[23949]: d32b04d294d30eed382adf7e696bae32612feb16e79fe2ba590c23be6d622ab7 2024-09-17T00:11:02.133 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:11:01 smithi002 systemd[1]: Started Ceph mgr.y for 3aad27a2-7489-11ef-bceb-c7b262605968. 2024-09-17T00:11:02.133 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:11:02 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: 2024-09-17T00:11:02.054+0000 7f628a641000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-17T00:11:02.133 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:11:02 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: 2024-09-17T00:11:02.135+0000 7f628a641000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-17T00:11:02.416 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:11:02 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: 2024-09-17T00:11:02.332+0000 7f628a641000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-17T00:11:02.604 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2024-09-17T00:11:02.604 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: { 2024-09-17T00:11:02.604 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "fsid": "3aad27a2-7489-11ef-bceb-c7b262605968", 2024-09-17T00:11:02.604 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "health": { 2024-09-17T00:11:02.604 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-09-17T00:11:02.604 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "checks": {}, 2024-09-17T00:11:02.604 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "mutes": [] 2024-09-17T00:11:02.604 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: }, 2024-09-17T00:11:02.604 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-09-17T00:11:02.605 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "quorum": [ 2024-09-17T00:11:02.605 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 0 2024-09-17T00:11:02.605 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: ], 2024-09-17T00:11:02.605 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "quorum_names": [ 2024-09-17T00:11:02.605 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "a" 2024-09-17T00:11:02.605 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: ], 2024-09-17T00:11:02.605 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "quorum_age": 5, 2024-09-17T00:11:02.605 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "monmap": { 2024-09-17T00:11:02.605 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "epoch": 1, 2024-09-17T00:11:02.605 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-09-17T00:11:02.605 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "num_mons": 1 2024-09-17T00:11:02.605 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: }, 2024-09-17T00:11:02.605 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "osdmap": { 2024-09-17T00:11:02.606 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "epoch": 1, 2024-09-17T00:11:02.606 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "num_osds": 0, 2024-09-17T00:11:02.606 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-09-17T00:11:02.606 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-09-17T00:11:02.606 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-09-17T00:11:02.606 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-09-17T00:11:02.606 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-09-17T00:11:02.606 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: }, 2024-09-17T00:11:02.606 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "pgmap": { 2024-09-17T00:11:02.606 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-09-17T00:11:02.606 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-09-17T00:11:02.606 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "num_pools": 0, 2024-09-17T00:11:02.606 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "num_objects": 0, 2024-09-17T00:11:02.607 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-09-17T00:11:02.607 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-09-17T00:11:02.607 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-09-17T00:11:02.607 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "bytes_total": 0 2024-09-17T00:11:02.607 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: }, 2024-09-17T00:11:02.607 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "fsmap": { 2024-09-17T00:11:02.607 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "epoch": 1, 2024-09-17T00:11:02.607 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "by_rank": [], 2024-09-17T00:11:02.607 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "up:standby": 0 2024-09-17T00:11:02.607 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: }, 2024-09-17T00:11:02.607 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "mgrmap": { 2024-09-17T00:11:02.607 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "available": false, 2024-09-17T00:11:02.607 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-09-17T00:11:02.608 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "modules": [ 2024-09-17T00:11:02.608 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "iostat", 2024-09-17T00:11:02.608 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "nfs", 2024-09-17T00:11:02.608 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "restful" 2024-09-17T00:11:02.608 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: ], 2024-09-17T00:11:02.608 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "services": {} 2024-09-17T00:11:02.608 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: }, 2024-09-17T00:11:02.608 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "servicemap": { 2024-09-17T00:11:02.608 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "epoch": 1, 2024-09-17T00:11:02.608 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "modified": "2024-09-17T00:10:48.443517+0000", 2024-09-17T00:11:02.608 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "services": {} 2024-09-17T00:11:02.608 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: }, 2024-09-17T00:11:02.608 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "progress_events": {} 2024-09-17T00:11:02.608 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: } 2024-09-17T00:11:02.729 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:11:02 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: 2024-09-17T00:11:02.423+0000 7f628a641000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-17T00:11:02.729 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:11:02 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: 2024-09-17T00:11:02.563+0000 7f628a641000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-17T00:11:03.142 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:11:02 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: 2024-09-17T00:11:02.790+0000 7f628a641000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-17T00:11:03.239 INFO:teuthology.orchestra.run.smithi002.stderr:mgr not available, waiting (1/15)... 2024-09-17T00:11:03.405 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:11:03 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: 2024-09-17T00:11:03.144+0000 7f628a641000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-17T00:11:03.405 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:11:03 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: 2024-09-17T00:11:03.235+0000 7f628a641000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-17T00:11:03.405 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:11:03 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: 2024-09-17T00:11:03.407+0000 7f628a641000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-17T00:11:03.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:11:03 smithi002 ceph-mon[23737]: from='client.? 172.21.15.2:0/2354210698' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-09-17T00:11:04.979 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:11:04 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: 2024-09-17T00:11:04.504+0000 7f628a641000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-17T00:11:04.979 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:11:04 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: 2024-09-17T00:11:04.598+0000 7f628a641000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-17T00:11:05.638 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:11:05 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: 2024-09-17T00:11:05.555+0000 7f628a641000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-17T00:11:05.717 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2024-09-17T00:11:05.717 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: { 2024-09-17T00:11:05.717 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "fsid": "3aad27a2-7489-11ef-bceb-c7b262605968", 2024-09-17T00:11:05.717 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "health": { 2024-09-17T00:11:05.717 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-09-17T00:11:05.717 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "checks": {}, 2024-09-17T00:11:05.717 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "mutes": [] 2024-09-17T00:11:05.717 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: }, 2024-09-17T00:11:05.717 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-09-17T00:11:05.717 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "quorum": [ 2024-09-17T00:11:05.717 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 0 2024-09-17T00:11:05.717 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: ], 2024-09-17T00:11:05.718 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "quorum_names": [ 2024-09-17T00:11:05.718 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "a" 2024-09-17T00:11:05.718 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: ], 2024-09-17T00:11:05.718 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "quorum_age": 8, 2024-09-17T00:11:05.718 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "monmap": { 2024-09-17T00:11:05.718 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "epoch": 1, 2024-09-17T00:11:05.718 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-09-17T00:11:05.718 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "num_mons": 1 2024-09-17T00:11:05.718 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: }, 2024-09-17T00:11:05.718 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "osdmap": { 2024-09-17T00:11:05.718 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "epoch": 1, 2024-09-17T00:11:05.718 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "num_osds": 0, 2024-09-17T00:11:05.718 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-09-17T00:11:05.718 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-09-17T00:11:05.719 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-09-17T00:11:05.719 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-09-17T00:11:05.719 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-09-17T00:11:05.719 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: }, 2024-09-17T00:11:05.719 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "pgmap": { 2024-09-17T00:11:05.719 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-09-17T00:11:05.719 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-09-17T00:11:05.719 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "num_pools": 0, 2024-09-17T00:11:05.719 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "num_objects": 0, 2024-09-17T00:11:05.719 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-09-17T00:11:05.719 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-09-17T00:11:05.719 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-09-17T00:11:05.719 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "bytes_total": 0 2024-09-17T00:11:05.719 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: }, 2024-09-17T00:11:05.720 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "fsmap": { 2024-09-17T00:11:05.720 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "epoch": 1, 2024-09-17T00:11:05.720 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "by_rank": [], 2024-09-17T00:11:05.720 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "up:standby": 0 2024-09-17T00:11:05.720 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: }, 2024-09-17T00:11:05.720 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "mgrmap": { 2024-09-17T00:11:05.720 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "available": false, 2024-09-17T00:11:05.720 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-09-17T00:11:05.720 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "modules": [ 2024-09-17T00:11:05.720 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "iostat", 2024-09-17T00:11:05.720 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "nfs", 2024-09-17T00:11:05.720 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "restful" 2024-09-17T00:11:05.720 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: ], 2024-09-17T00:11:05.720 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "services": {} 2024-09-17T00:11:05.721 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: }, 2024-09-17T00:11:05.721 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "servicemap": { 2024-09-17T00:11:05.721 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "epoch": 1, 2024-09-17T00:11:05.721 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "modified": "2024-09-17T00:10:48.443517+0000", 2024-09-17T00:11:05.721 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "services": {} 2024-09-17T00:11:05.721 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: }, 2024-09-17T00:11:05.721 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "progress_events": {} 2024-09-17T00:11:05.721 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: } 2024-09-17T00:11:05.893 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:11:05 smithi002 ceph-mon[23737]: from='client.? 172.21.15.2:0/942162119' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-09-17T00:11:05.893 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:11:05 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: 2024-09-17T00:11:05.885+0000 7f628a641000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-17T00:11:06.229 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:11:06 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: 2024-09-17T00:11:06.031+0000 7f628a641000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-17T00:11:06.229 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:11:06 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: 2024-09-17T00:11:06.128+0000 7f628a641000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-17T00:11:06.412 INFO:teuthology.orchestra.run.smithi002.stderr:mgr not available, waiting (2/15)... 2024-09-17T00:11:06.481 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:11:06 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: 2024-09-17T00:11:06.233+0000 7f628a641000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-17T00:11:06.481 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:11:06 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: 2024-09-17T00:11:06.327+0000 7f628a641000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-17T00:11:06.482 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:11:06 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: 2024-09-17T00:11:06.484+0000 7f628a641000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-17T00:11:06.885 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:11:06 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: 2024-09-17T00:11:06.581+0000 7f628a641000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-17T00:11:06.885 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:11:06 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: 2024-09-17T00:11:06.681+0000 7f628a641000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-17T00:11:07.156 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:11:06 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: 2024-09-17T00:11:06.977+0000 7f628a641000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-17T00:11:07.156 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:11:07 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: 2024-09-17T00:11:07.067+0000 7f628a641000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-17T00:11:07.156 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:11:07 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: 2024-09-17T00:11:07.158+0000 7f628a641000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-17T00:11:08.535 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:11:08 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: 2024-09-17T00:11:08.268+0000 7f628a641000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-17T00:11:08.535 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:11:08 smithi002 ceph-mon[23737]: Activating manager daemon y 2024-09-17T00:11:08.535 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:11:08 smithi002 ceph-mon[23737]: mgrmap e2: y(active, starting, since 0.00291359s) 2024-09-17T00:11:08.536 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:11:08 smithi002 ceph-mon[23737]: from='mgr.14100 172.21.15.2:0/2838505675' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-17T00:11:08.536 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:11:08 smithi002 ceph-mon[23737]: from='mgr.14100 172.21.15.2:0/2838505675' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-17T00:11:08.536 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:11:08 smithi002 ceph-mon[23737]: from='mgr.14100 172.21.15.2:0/2838505675' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-17T00:11:08.536 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:11:08 smithi002 ceph-mon[23737]: from='mgr.14100 172.21.15.2:0/2838505675' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T00:11:08.536 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:11:08 smithi002 ceph-mon[23737]: from='mgr.14100 172.21.15.2:0/2838505675' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-17T00:11:08.536 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:11:08 smithi002 ceph-mon[23737]: Manager daemon y is now available 2024-09-17T00:11:08.536 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:11:08 smithi002 ceph-mon[23737]: from='mgr.14100 172.21.15.2:0/2838505675' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T00:11:08.536 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:11:08 smithi002 ceph-mon[23737]: from='mgr.14100 172.21.15.2:0/2838505675' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T00:11:08.536 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:11:08 smithi002 ceph-mon[23737]: from='mgr.14100 172.21.15.2:0/2838505675' entity='mgr.y' 2024-09-17T00:11:08.536 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:11:08 smithi002 ceph-mon[23737]: from='mgr.14100 172.21.15.2:0/2838505675' entity='mgr.y' 2024-09-17T00:11:08.537 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:11:08 smithi002 ceph-mon[23737]: from='mgr.14100 172.21.15.2:0/2838505675' entity='mgr.y' 2024-09-17T00:11:08.901 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2024-09-17T00:11:08.901 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: { 2024-09-17T00:11:08.901 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "fsid": "3aad27a2-7489-11ef-bceb-c7b262605968", 2024-09-17T00:11:08.901 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "health": { 2024-09-17T00:11:08.901 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-09-17T00:11:08.901 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "checks": {}, 2024-09-17T00:11:08.901 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "mutes": [] 2024-09-17T00:11:08.901 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: }, 2024-09-17T00:11:08.901 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-09-17T00:11:08.902 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "quorum": [ 2024-09-17T00:11:08.902 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 0 2024-09-17T00:11:08.902 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: ], 2024-09-17T00:11:08.902 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "quorum_names": [ 2024-09-17T00:11:08.902 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "a" 2024-09-17T00:11:08.902 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: ], 2024-09-17T00:11:08.902 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "quorum_age": 11, 2024-09-17T00:11:08.902 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "monmap": { 2024-09-17T00:11:08.902 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "epoch": 1, 2024-09-17T00:11:08.902 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-09-17T00:11:08.902 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "num_mons": 1 2024-09-17T00:11:08.902 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: }, 2024-09-17T00:11:08.902 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "osdmap": { 2024-09-17T00:11:08.902 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "epoch": 1, 2024-09-17T00:11:08.902 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "num_osds": 0, 2024-09-17T00:11:08.903 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-09-17T00:11:08.903 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-09-17T00:11:08.906 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-09-17T00:11:08.906 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-09-17T00:11:08.906 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-09-17T00:11:08.906 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: }, 2024-09-17T00:11:08.906 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "pgmap": { 2024-09-17T00:11:08.906 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-09-17T00:11:08.906 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-09-17T00:11:08.906 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "num_pools": 0, 2024-09-17T00:11:08.906 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "num_objects": 0, 2024-09-17T00:11:08.907 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-09-17T00:11:08.907 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-09-17T00:11:08.907 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-09-17T00:11:08.907 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "bytes_total": 0 2024-09-17T00:11:08.907 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: }, 2024-09-17T00:11:08.907 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "fsmap": { 2024-09-17T00:11:08.907 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "epoch": 1, 2024-09-17T00:11:08.907 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "by_rank": [], 2024-09-17T00:11:08.907 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "up:standby": 0 2024-09-17T00:11:08.907 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: }, 2024-09-17T00:11:08.907 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "mgrmap": { 2024-09-17T00:11:08.907 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "available": false, 2024-09-17T00:11:08.907 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-09-17T00:11:08.907 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "modules": [ 2024-09-17T00:11:08.907 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "iostat", 2024-09-17T00:11:08.908 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "nfs", 2024-09-17T00:11:08.908 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "restful" 2024-09-17T00:11:08.908 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: ], 2024-09-17T00:11:08.908 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "services": {} 2024-09-17T00:11:08.908 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: }, 2024-09-17T00:11:08.908 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "servicemap": { 2024-09-17T00:11:08.908 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "epoch": 1, 2024-09-17T00:11:08.908 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "modified": "2024-09-17T00:10:48.443517+0000", 2024-09-17T00:11:08.908 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "services": {} 2024-09-17T00:11:08.908 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: }, 2024-09-17T00:11:08.908 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "progress_events": {} 2024-09-17T00:11:08.908 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: } 2024-09-17T00:11:09.476 INFO:teuthology.orchestra.run.smithi002.stderr:mgr not available, waiting (3/15)... 2024-09-17T00:11:09.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:11:09 smithi002 ceph-mon[23737]: from='client.? 172.21.15.2:0/19887390' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-09-17T00:11:09.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:11:09 smithi002 ceph-mon[23737]: mgrmap e3: y(active, since 1.00627s) 2024-09-17T00:11:11.784 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:11:11 smithi002 ceph-mon[23737]: mgrmap e4: y(active, since 2s) 2024-09-17T00:11:12.147 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2024-09-17T00:11:12.147 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: { 2024-09-17T00:11:12.147 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "fsid": "3aad27a2-7489-11ef-bceb-c7b262605968", 2024-09-17T00:11:12.148 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "health": { 2024-09-17T00:11:12.148 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-09-17T00:11:12.148 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "checks": {}, 2024-09-17T00:11:12.148 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "mutes": [] 2024-09-17T00:11:12.148 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: }, 2024-09-17T00:11:12.148 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-09-17T00:11:12.148 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "quorum": [ 2024-09-17T00:11:12.148 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 0 2024-09-17T00:11:12.148 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: ], 2024-09-17T00:11:12.148 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "quorum_names": [ 2024-09-17T00:11:12.148 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "a" 2024-09-17T00:11:12.148 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: ], 2024-09-17T00:11:12.148 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "quorum_age": 14, 2024-09-17T00:11:12.148 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "monmap": { 2024-09-17T00:11:12.149 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "epoch": 1, 2024-09-17T00:11:12.149 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-09-17T00:11:12.149 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "num_mons": 1 2024-09-17T00:11:12.149 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: }, 2024-09-17T00:11:12.149 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "osdmap": { 2024-09-17T00:11:12.149 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "epoch": 1, 2024-09-17T00:11:12.149 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "num_osds": 0, 2024-09-17T00:11:12.149 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-09-17T00:11:12.149 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-09-17T00:11:12.149 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-09-17T00:11:12.149 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-09-17T00:11:12.149 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-09-17T00:11:12.153 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: }, 2024-09-17T00:11:12.153 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "pgmap": { 2024-09-17T00:11:12.153 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-09-17T00:11:12.153 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-09-17T00:11:12.153 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "num_pools": 0, 2024-09-17T00:11:12.153 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "num_objects": 0, 2024-09-17T00:11:12.153 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-09-17T00:11:12.153 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-09-17T00:11:12.153 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-09-17T00:11:12.153 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "bytes_total": 0 2024-09-17T00:11:12.153 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: }, 2024-09-17T00:11:12.153 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "fsmap": { 2024-09-17T00:11:12.153 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "epoch": 1, 2024-09-17T00:11:12.153 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "by_rank": [], 2024-09-17T00:11:12.154 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "up:standby": 0 2024-09-17T00:11:12.154 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: }, 2024-09-17T00:11:12.154 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "mgrmap": { 2024-09-17T00:11:12.154 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "available": true, 2024-09-17T00:11:12.154 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-09-17T00:11:12.154 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "modules": [ 2024-09-17T00:11:12.154 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "iostat", 2024-09-17T00:11:12.154 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "nfs", 2024-09-17T00:11:12.154 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "restful" 2024-09-17T00:11:12.154 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: ], 2024-09-17T00:11:12.154 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "services": {} 2024-09-17T00:11:12.154 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: }, 2024-09-17T00:11:12.154 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "servicemap": { 2024-09-17T00:11:12.154 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "epoch": 1, 2024-09-17T00:11:12.154 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "modified": "2024-09-17T00:10:48.443517+0000", 2024-09-17T00:11:12.155 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "services": {} 2024-09-17T00:11:12.155 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: }, 2024-09-17T00:11:12.155 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "progress_events": {} 2024-09-17T00:11:12.155 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: } 2024-09-17T00:11:12.723 INFO:teuthology.orchestra.run.smithi002.stderr:mgr is available 2024-09-17T00:11:12.846 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:11:12 smithi002 ceph-mon[23737]: from='client.? 172.21.15.2:0/3731101578' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-09-17T00:11:13.345 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2024-09-17T00:11:13.345 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: [global] 2024-09-17T00:11:13.345 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: fsid = 3aad27a2-7489-11ef-bceb-c7b262605968 2024-09-17T00:11:13.345 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-09-17T00:11:13.345 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-09-17T00:11:13.345 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-09-17T00:11:13.346 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-09-17T00:11:13.346 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2024-09-17T00:11:13.346 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: [mgr] 2024-09-17T00:11:13.346 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-09-17T00:11:13.346 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 2024-09-17T00:11:13.346 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: [osd] 2024-09-17T00:11:13.346 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-09-17T00:11:13.346 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-09-17T00:11:13.893 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:11:13 smithi002 ceph-mon[23737]: from='client.? 172.21.15.2:0/1473743755' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-09-17T00:11:13.894 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:11:13 smithi002 ceph-mon[23737]: from='client.? 172.21.15.2:0/1473743755' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-09-17T00:11:13.896 INFO:teuthology.orchestra.run.smithi002.stderr:Enabling cephadm module... 2024-09-17T00:11:14.941 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:11:14 smithi002 ceph-mon[23737]: from='client.? 172.21.15.2:0/3443550135' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-09-17T00:11:14.941 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:11:14 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: ignoring --setuser ceph since I am not root 2024-09-17T00:11:14.941 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:11:14 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: ignoring --setgroup ceph since I am not root 2024-09-17T00:11:14.941 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:11:14 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: 2024-09-17T00:11:14.778+0000 7f8b2b5ff000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-17T00:11:14.941 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:11:14 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: 2024-09-17T00:11:14.859+0000 7f8b2b5ff000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-17T00:11:15.229 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:11:15 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: 2024-09-17T00:11:15.050+0000 7f8b2b5ff000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-17T00:11:15.229 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:11:15 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: 2024-09-17T00:11:15.140+0000 7f8b2b5ff000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-17T00:11:15.611 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:11:15 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: 2024-09-17T00:11:15.278+0000 7f8b2b5ff000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-17T00:11:15.612 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:11:15 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: 2024-09-17T00:11:15.499+0000 7f8b2b5ff000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-17T00:11:15.939 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:11:15 smithi002 ceph-mon[23737]: from='client.? 172.21.15.2:0/3443550135' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-09-17T00:11:15.939 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:11:15 smithi002 ceph-mon[23737]: mgrmap e5: y(active, since 6s) 2024-09-17T00:11:15.939 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:11:15 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: 2024-09-17T00:11:15.849+0000 7f8b2b5ff000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-17T00:11:15.939 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:11:15 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: 2024-09-17T00:11:15.941+0000 7f8b2b5ff000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-17T00:11:15.953 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: { 2024-09-17T00:11:15.953 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "epoch": 5, 2024-09-17T00:11:15.953 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "available": true, 2024-09-17T00:11:15.953 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "active_name": "y", 2024-09-17T00:11:15.953 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "num_standby": 0 2024-09-17T00:11:15.954 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: } 2024-09-17T00:11:16.229 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:11:16 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: 2024-09-17T00:11:16.116+0000 7f8b2b5ff000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-17T00:11:16.576 INFO:teuthology.orchestra.run.smithi002.stderr:Waiting for the mgr to restart... 2024-09-17T00:11:16.576 INFO:teuthology.orchestra.run.smithi002.stderr:Waiting for mgr epoch 5... 2024-09-17T00:11:16.884 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:11:16 smithi002 ceph-mon[23737]: from='client.? 172.21.15.2:0/2125917457' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-09-17T00:11:17.729 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:11:17 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: 2024-09-17T00:11:17.235+0000 7f8b2b5ff000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-17T00:11:17.729 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:11:17 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: 2024-09-17T00:11:17.328+0000 7f8b2b5ff000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-17T00:11:18.579 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:11:18 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: 2024-09-17T00:11:18.265+0000 7f8b2b5ff000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-17T00:11:18.580 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:11:18 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: 2024-09-17T00:11:18.582+0000 7f8b2b5ff000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-17T00:11:18.910 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:11:18 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: 2024-09-17T00:11:18.720+0000 7f8b2b5ff000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-17T00:11:18.910 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:11:18 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: 2024-09-17T00:11:18.811+0000 7f8b2b5ff000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-17T00:11:18.910 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:11:18 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: 2024-09-17T00:11:18.912+0000 7f8b2b5ff000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-17T00:11:19.229 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:11:19 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: 2024-09-17T00:11:19.003+0000 7f8b2b5ff000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-17T00:11:19.229 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:11:19 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: 2024-09-17T00:11:19.155+0000 7f8b2b5ff000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-17T00:11:19.642 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:11:19 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: 2024-09-17T00:11:19.251+0000 7f8b2b5ff000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-17T00:11:19.642 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:11:19 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: 2024-09-17T00:11:19.351+0000 7f8b2b5ff000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-17T00:11:19.642 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:11:19 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: 2024-09-17T00:11:19.645+0000 7f8b2b5ff000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-17T00:11:19.979 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:11:19 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: 2024-09-17T00:11:19.736+0000 7f8b2b5ff000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-17T00:11:19.979 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:11:19 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: 2024-09-17T00:11:19.827+0000 7f8b2b5ff000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-17T00:11:21.229 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:11:20 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: 2024-09-17T00:11:20.936+0000 7f8b2b5ff000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-17T00:11:22.000 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: { 2024-09-17T00:11:22.000 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "mgrmap_epoch": 7, 2024-09-17T00:11:22.000 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "initialized": true 2024-09-17T00:11:22.000 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: } 2024-09-17T00:11:22.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:11:21 smithi002 ceph-mon[23737]: Active manager daemon y restarted 2024-09-17T00:11:22.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:11:21 smithi002 ceph-mon[23737]: Activating manager daemon y 2024-09-17T00:11:22.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:11:21 smithi002 ceph-mon[23737]: osdmap e2: 0 total, 0 up, 0 in 2024-09-17T00:11:22.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:11:21 smithi002 ceph-mon[23737]: mgrmap e6: y(active, starting, since 0.0522594s) 2024-09-17T00:11:22.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:11:21 smithi002 ceph-mon[23737]: from='mgr.14120 172.21.15.2:0/1914495088' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T00:11:22.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:11:21 smithi002 ceph-mon[23737]: from='mgr.14120 172.21.15.2:0/1914495088' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-17T00:11:22.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:11:21 smithi002 ceph-mon[23737]: from='mgr.14120 172.21.15.2:0/1914495088' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-17T00:11:22.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:11:21 smithi002 ceph-mon[23737]: from='mgr.14120 172.21.15.2:0/1914495088' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-17T00:11:22.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:11:21 smithi002 ceph-mon[23737]: from='mgr.14120 172.21.15.2:0/1914495088' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-17T00:11:22.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:11:21 smithi002 ceph-mon[23737]: Manager daemon y is now available 2024-09-17T00:11:22.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:11:21 smithi002 ceph-mon[23737]: from='mgr.14120 172.21.15.2:0/1914495088' entity='mgr.y' 2024-09-17T00:11:22.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:11:21 smithi002 ceph-mon[23737]: from='mgr.14120 172.21.15.2:0/1914495088' entity='mgr.y' 2024-09-17T00:11:22.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:11:21 smithi002 ceph-mon[23737]: from='mgr.14120 172.21.15.2:0/1914495088' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T00:11:22.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:11:21 smithi002 ceph-mon[23737]: from='mgr.14120 172.21.15.2:0/1914495088' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T00:11:22.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:11:21 smithi002 ceph-mon[23737]: from='mgr.14120 172.21.15.2:0/1914495088' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T00:11:22.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:11:21 smithi002 ceph-mon[23737]: from='mgr.14120 172.21.15.2:0/1914495088' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T00:11:22.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:11:21 smithi002 ceph-mon[23737]: from='mgr.14120 172.21.15.2:0/1914495088' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T00:11:22.597 INFO:teuthology.orchestra.run.smithi002.stderr:mgr epoch 5 is available 2024-09-17T00:11:22.598 INFO:teuthology.orchestra.run.smithi002.stderr:Setting orchestrator backend to cephadm... 2024-09-17T00:11:23.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:11:22 smithi002 ceph-mon[23737]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-09-17T00:11:23.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:11:22 smithi002 ceph-mon[23737]: mgrmap e7: y(active, since 1.05465s) 2024-09-17T00:11:23.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:11:22 smithi002 ceph-mon[23737]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-09-17T00:11:24.151 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:11:24 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: [17/Sep/2024:00:11:24] ENGINE Bus STARTING 2024-09-17T00:11:24.479 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:11:24 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: [17/Sep/2024:00:11:24] ENGINE Serving on https://172.21.15.2:7150 2024-09-17T00:11:24.479 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:11:24 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: [17/Sep/2024:00:11:24] ENGINE Bus STARTED 2024-09-17T00:11:24.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:11:24 smithi002 ceph-mon[23737]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:11:24.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:11:24 smithi002 ceph-mon[23737]: from='mgr.14120 172.21.15.2:0/1914495088' entity='mgr.y' 2024-09-17T00:11:24.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:11:24 smithi002 ceph-mon[23737]: from='mgr.14120 172.21.15.2:0/1914495088' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T00:11:24.480 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:11:24 smithi002 ceph-mon[23737]: mgrmap e8: y(active, since 2s) 2024-09-17T00:11:24.480 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:11:24 smithi002 ceph-mon[23737]: from='mgr.14120 172.21.15.2:0/1914495088' entity='mgr.y' 2024-09-17T00:11:24.480 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:11:24 smithi002 ceph-mon[23737]: from='mgr.14120 172.21.15.2:0/1914495088' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T00:11:24.542 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: value unchanged 2024-09-17T00:11:25.135 INFO:teuthology.orchestra.run.smithi002.stderr:Generating ssh key... 2024-09-17T00:11:25.543 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:11:25 smithi002 ceph-mon[23737]: [17/Sep/2024:00:11:24] ENGINE Bus STARTING 2024-09-17T00:11:25.544 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:11:25 smithi002 ceph-mon[23737]: [17/Sep/2024:00:11:24] ENGINE Serving on https://172.21.15.2:7150 2024-09-17T00:11:25.544 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:11:25 smithi002 ceph-mon[23737]: [17/Sep/2024:00:11:24] ENGINE Bus STARTED 2024-09-17T00:11:25.544 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:11:25 smithi002 ceph-mon[23737]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:11:26.914 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:11:26 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: Generating public/private rsa key pair. 2024-09-17T00:11:26.914 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:11:26 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: Your identification has been saved in /tmp/tmpu41yc7qh/key. 2024-09-17T00:11:26.914 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:11:26 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: Your public key has been saved in /tmp/tmpu41yc7qh/key.pub. 2024-09-17T00:11:26.914 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:11:26 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: The key fingerprint is: 2024-09-17T00:11:26.914 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:11:26 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: SHA256:OFy4ZW8cQISU/cABIakxDWRRm5kMnBjXOYDlxG25aGU ceph-3aad27a2-7489-11ef-bceb-c7b262605968 2024-09-17T00:11:26.914 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:11:26 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: The key's randomart image is: 2024-09-17T00:11:26.914 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:11:26 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: +---[RSA 3072]----+ 2024-09-17T00:11:26.915 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:11:26 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: |.O@Xo=+X=. | 2024-09-17T00:11:26.915 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:11:26 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: |o=*oE*o.+. | 2024-09-17T00:11:26.915 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:11:26 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: | .O*o. +o. | 2024-09-17T00:11:26.915 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:11:26 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: | + .. * o.. | 2024-09-17T00:11:26.915 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:11:26 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: | . = S + | 2024-09-17T00:11:26.915 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:11:26 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: | . . | 2024-09-17T00:11:26.915 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:11:26 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: | | 2024-09-17T00:11:26.915 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:11:26 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: | | 2024-09-17T00:11:26.915 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:11:26 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: | | 2024-09-17T00:11:26.915 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:11:26 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: +----[SHA256]-----+ 2024-09-17T00:11:27.871 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:11:27 smithi002 ceph-mon[23737]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:11:27.871 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:11:27 smithi002 ceph-mon[23737]: Generating ssh key... 2024-09-17T00:11:27.871 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:11:27 smithi002 ceph-mon[23737]: from='mgr.14120 172.21.15.2:0/1914495088' entity='mgr.y' 2024-09-17T00:11:27.871 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:11:27 smithi002 ceph-mon[23737]: from='mgr.14120 172.21.15.2:0/1914495088' entity='mgr.y' 2024-09-17T00:11:27.872 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCY/LAtbkiw9PU/IdeC4kdvvnE6A1dntm01o3RQJIeQ2ZiT3I+gDKlS/btW0o7gnB0xAUtQ/jyPRRSE+W14yDF7sSQHIYNce//9gSM5ONgn2JuwYHR2O0UJmKdYuMT4+3PSIGDgJ92MICkdfLYX9pdN1bZnA7fw2g2TWV+birej94tOtvrHDQEG7Q3Hr1E8xyCd/Cnr2Ap3yqKxAoV11EdXPZQvIt7VlTFdGQ4ibpJhdtX2tmj6bi/FDbo34V6SRhfYAJa8BD5dhGzc8i/olx3YNFa9xdii+tHBhsmys1w4uhRpb+gG3B8l9ZIHNVbHHs/xE8AWCFYM45e8Zc+m0TJcKJx7JEMUczq6p3971XNXFPCbJLqOCZG7FDskEBNXFiK0rdJbeXa55m8osFppuN595INtZj/8FqMI3KKcO3iMlcFXLceladxQtSRHLRdJSXkd3bDvJrC7OuyX66Yg5ZRx1mGTHp8YKTI7vULmuIbt+ORlZgyAfzvuSUMjawXYq0s= ceph-3aad27a2-7489-11ef-bceb-c7b262605968 2024-09-17T00:11:28.465 INFO:teuthology.orchestra.run.smithi002.stderr:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-09-17T00:11:28.465 INFO:teuthology.orchestra.run.smithi002.stderr:Adding key to root@localhost authorized_keys... 2024-09-17T00:11:28.479 INFO:teuthology.orchestra.run.smithi002.stderr:Adding host smithi002... 2024-09-17T00:11:28.713 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:11:28 smithi002 ceph-mon[23737]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:11:29.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:11:29 smithi002 ceph-mon[23737]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi002", "addr": "172.21.15.2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:11:29.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:11:29 smithi002 ceph-mon[23737]: Deploying cephadm binary to smithi002 2024-09-17T00:11:30.023 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: Added host 'smithi002' with addr '172.21.15.2' 2024-09-17T00:11:30.622 INFO:teuthology.orchestra.run.smithi002.stderr:Deploying unmanaged mon service... 2024-09-17T00:11:31.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:11:31 smithi002 ceph-mon[23737]: from='mgr.14120 172.21.15.2:0/1914495088' entity='mgr.y' 2024-09-17T00:11:31.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:11:31 smithi002 ceph-mon[23737]: Added host smithi002 2024-09-17T00:11:31.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:11:31 smithi002 ceph-mon[23737]: from='mgr.14120 172.21.15.2:0/1914495088' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T00:11:31.280 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: Scheduled mon update... 2024-09-17T00:11:32.164 INFO:teuthology.orchestra.run.smithi002.stderr:Deploying unmanaged mgr service... 2024-09-17T00:11:32.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:11:32 smithi002 ceph-mon[23737]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:11:32.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:11:32 smithi002 ceph-mon[23737]: Saving service mon spec with placement count:5 2024-09-17T00:11:32.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:11:32 smithi002 ceph-mon[23737]: from='mgr.14120 172.21.15.2:0/1914495088' entity='mgr.y' 2024-09-17T00:11:32.828 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: Scheduled mgr update... 2024-09-17T00:11:34.005 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:11:33 smithi002 ceph-mon[23737]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:11:34.005 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:11:33 smithi002 ceph-mon[23737]: Saving service mgr spec with placement count:2 2024-09-17T00:11:34.005 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:11:33 smithi002 ceph-mon[23737]: from='mgr.14120 172.21.15.2:0/1914495088' entity='mgr.y' 2024-09-17T00:11:35.431 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:11:35 smithi002 ceph-mon[23737]: from='client.? 172.21.15.2:0/2449187265' entity='client.admin' 2024-09-17T00:11:35.431 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:11:35 smithi002 ceph-mon[23737]: from='mgr.14120 172.21.15.2:0/1914495088' entity='mgr.y' 2024-09-17T00:11:36.397 INFO:teuthology.orchestra.run.smithi002.stderr:Enabling the dashboard module... 2024-09-17T00:11:36.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:11:36 smithi002 ceph-mon[23737]: from='mgr.14120 172.21.15.2:0/1914495088' entity='mgr.y' 2024-09-17T00:11:36.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:11:36 smithi002 ceph-mon[23737]: from='client.? 172.21.15.2:0/1138165033' entity='client.admin' 2024-09-17T00:11:37.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:11:37 smithi002 ceph-mon[23737]: from='client.? 172.21.15.2:0/2353062550' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-09-17T00:11:37.980 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:11:37 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: ignoring --setuser ceph since I am not root 2024-09-17T00:11:37.980 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:11:37 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: ignoring --setgroup ceph since I am not root 2024-09-17T00:11:37.980 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:11:37 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: 2024-09-17T00:11:37.901+0000 7f1e4848f000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-17T00:11:38.269 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:11:37 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: 2024-09-17T00:11:37.983+0000 7f1e4848f000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-17T00:11:38.269 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:11:38 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: 2024-09-17T00:11:38.179+0000 7f1e4848f000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-17T00:11:38.269 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:11:38 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: 2024-09-17T00:11:38.271+0000 7f1e4848f000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-17T00:11:38.539 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:11:38 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: 2024-09-17T00:11:38.407+0000 7f1e4848f000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-17T00:11:38.848 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:11:38 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: 2024-09-17T00:11:38.630+0000 7f1e4848f000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-17T00:11:38.849 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:11:38 smithi002 ceph-mon[23737]: from='client.? 172.21.15.2:0/2353062550' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-09-17T00:11:38.849 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:11:38 smithi002 ceph-mon[23737]: mgrmap e9: y(active, since 16s) 2024-09-17T00:11:39.127 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:11:38 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: 2024-09-17T00:11:38.982+0000 7f1e4848f000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-17T00:11:39.127 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:11:39 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: 2024-09-17T00:11:39.074+0000 7f1e4848f000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-17T00:11:39.128 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: { 2024-09-17T00:11:39.128 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "epoch": 9, 2024-09-17T00:11:39.129 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "available": true, 2024-09-17T00:11:39.129 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "active_name": "y", 2024-09-17T00:11:39.129 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "num_standby": 0 2024-09-17T00:11:39.129 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: } 2024-09-17T00:11:39.435 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:11:39 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: 2024-09-17T00:11:39.249+0000 7f1e4848f000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-17T00:11:39.721 INFO:teuthology.orchestra.run.smithi002.stderr:Waiting for the mgr to restart... 2024-09-17T00:11:39.722 INFO:teuthology.orchestra.run.smithi002.stderr:Waiting for mgr epoch 9... 2024-09-17T00:11:39.969 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:11:39 smithi002 ceph-mon[23737]: from='client.? 172.21.15.2:0/2457209316' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-09-17T00:11:40.729 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:11:40 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: 2024-09-17T00:11:40.362+0000 7f1e4848f000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-17T00:11:40.729 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:11:40 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: 2024-09-17T00:11:40.456+0000 7f1e4848f000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-17T00:11:41.718 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:11:41 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: 2024-09-17T00:11:41.400+0000 7f1e4848f000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-17T00:11:41.979 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:11:41 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: 2024-09-17T00:11:41.720+0000 7f1e4848f000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-17T00:11:41.979 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:11:41 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: 2024-09-17T00:11:41.860+0000 7f1e4848f000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-17T00:11:41.979 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:11:41 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: 2024-09-17T00:11:41.952+0000 7f1e4848f000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-17T00:11:42.394 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:11:42 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: 2024-09-17T00:11:42.054+0000 7f1e4848f000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-17T00:11:42.394 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:11:42 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: 2024-09-17T00:11:42.145+0000 7f1e4848f000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-17T00:11:42.394 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:11:42 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: 2024-09-17T00:11:42.299+0000 7f1e4848f000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-17T00:11:42.394 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:11:42 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: 2024-09-17T00:11:42.396+0000 7f1e4848f000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-17T00:11:42.729 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:11:42 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: 2024-09-17T00:11:42.497+0000 7f1e4848f000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-17T00:11:43.229 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:11:42 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: 2024-09-17T00:11:42.793+0000 7f1e4848f000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-17T00:11:43.229 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:11:42 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: 2024-09-17T00:11:42.885+0000 7f1e4848f000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-17T00:11:43.229 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:11:42 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: 2024-09-17T00:11:42.978+0000 7f1e4848f000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-17T00:11:44.341 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:11:44 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: 2024-09-17T00:11:44.091+0000 7f1e4848f000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-17T00:11:44.689 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:11:44 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: [17/Sep/2024:00:11:44] ENGINE Bus STARTING 2024-09-17T00:11:44.979 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:11:44 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: [17/Sep/2024:00:11:44] ENGINE Serving on https://172.21.15.2:7150 2024-09-17T00:11:44.979 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:11:44 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: [17/Sep/2024:00:11:44] ENGINE Bus STARTED 2024-09-17T00:11:45.153 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: { 2024-09-17T00:11:45.153 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "mgrmap_epoch": 11, 2024-09-17T00:11:45.153 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: "initialized": true 2024-09-17T00:11:45.153 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: } 2024-09-17T00:11:45.478 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:11:45 smithi002 ceph-mon[23737]: Active manager daemon y restarted 2024-09-17T00:11:45.478 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:11:45 smithi002 ceph-mon[23737]: Activating manager daemon y 2024-09-17T00:11:45.478 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:11:45 smithi002 ceph-mon[23737]: osdmap e3: 0 total, 0 up, 0 in 2024-09-17T00:11:45.478 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:11:45 smithi002 ceph-mon[23737]: mgrmap e10: y(active, starting, since 0.0523694s) 2024-09-17T00:11:45.478 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:11:45 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T00:11:45.478 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:11:45 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-17T00:11:45.478 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:11:45 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-17T00:11:45.478 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:11:45 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-17T00:11:45.478 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:11:45 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-17T00:11:45.478 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:11:45 smithi002 ceph-mon[23737]: Manager daemon y is now available 2024-09-17T00:11:45.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:11:45 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T00:11:45.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:11:45 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T00:11:45.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:11:45 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T00:11:45.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:11:45 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T00:11:45.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:11:45 smithi002 ceph-mon[23737]: [17/Sep/2024:00:11:44] ENGINE Bus STARTING 2024-09-17T00:11:45.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:11:45 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:11:45.781 INFO:teuthology.orchestra.run.smithi002.stderr:mgr epoch 9 is available 2024-09-17T00:11:45.781 INFO:teuthology.orchestra.run.smithi002.stderr:Generating a dashboard self-signed certificate... 2024-09-17T00:11:46.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:11:46 smithi002 ceph-mon[23737]: [17/Sep/2024:00:11:44] ENGINE Serving on https://172.21.15.2:7150 2024-09-17T00:11:46.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:11:46 smithi002 ceph-mon[23737]: [17/Sep/2024:00:11:44] ENGINE Bus STARTED 2024-09-17T00:11:46.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:11:46 smithi002 ceph-mon[23737]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-09-17T00:11:46.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:11:46 smithi002 ceph-mon[23737]: mgrmap e11: y(active, since 1.05552s) 2024-09-17T00:11:46.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:11:46 smithi002 ceph-mon[23737]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-09-17T00:11:46.541 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: Self-signed certificate created 2024-09-17T00:11:47.203 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:11:47 smithi002 ceph-mon[23737]: mgrmap e12: y(active, since 2s) 2024-09-17T00:11:47.204 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:11:47 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:11:47.204 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:11:47 smithi002 ceph-mon[23737]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:11:47.204 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:11:47 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:11:47.204 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:11:47 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:11:47.205 INFO:teuthology.orchestra.run.smithi002.stderr:Creating initial admin user... 2024-09-17T00:11:48.131 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: {"username": "admin", "password": "$2b$12$/KbmMZb9yc9T1LaYh3i8Uu0TGdg0SFBNcJmqZYuVSresqADJJDHwe", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1726531908, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-09-17T00:11:48.946 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:11:48 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:11:48.947 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:11:48 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:11:49.233 INFO:teuthology.orchestra.run.smithi002.stderr:Fetching dashboard port number... 2024-09-17T00:11:49.977 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:11:49 smithi002 ceph-mon[23737]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:11:50.183 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: 8443 2024-09-17T00:11:50.772 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:11:50 smithi002 ceph-mon[23737]: from='client.? 172.21.15.2:0/586062116' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-09-17T00:11:50.785 INFO:teuthology.orchestra.run.smithi002.stderr:systemctl: disabled 2024-09-17T00:11:50.793 INFO:teuthology.orchestra.run.smithi002.stderr:systemctl: inactive 2024-09-17T00:11:50.794 INFO:teuthology.orchestra.run.smithi002.stderr:firewalld.service is not enabled 2024-09-17T00:11:50.794 INFO:teuthology.orchestra.run.smithi002.stderr:Not possible to open ports <[8443]>. firewalld.service is not available 2024-09-17T00:11:50.797 INFO:teuthology.orchestra.run.smithi002.stderr:Ceph Dashboard is now available at: 2024-09-17T00:11:50.797 INFO:teuthology.orchestra.run.smithi002.stderr: 2024-09-17T00:11:50.797 INFO:teuthology.orchestra.run.smithi002.stderr: URL: https://smithi002.front.sepia.ceph.com:8443/ 2024-09-17T00:11:50.797 INFO:teuthology.orchestra.run.smithi002.stderr: User: admin 2024-09-17T00:11:50.797 INFO:teuthology.orchestra.run.smithi002.stderr: Password: p7qjs5p4pu 2024-09-17T00:11:50.797 INFO:teuthology.orchestra.run.smithi002.stderr: 2024-09-17T00:11:50.797 INFO:teuthology.orchestra.run.smithi002.stderr:Enabling autotune for osd_memory_target 2024-09-17T00:11:51.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:11:51 smithi002 ceph-mon[23737]: mgrmap e13: y(active, since 6s) 2024-09-17T00:11:52.797 INFO:teuthology.orchestra.run.smithi002.stderr:/usr/bin/ceph: set mgr/dashboard/cluster/status 2024-09-17T00:11:53.346 INFO:teuthology.orchestra.run.smithi002.stderr:You can access the Ceph CLI with: 2024-09-17T00:11:53.346 INFO:teuthology.orchestra.run.smithi002.stderr: 2024-09-17T00:11:53.346 INFO:teuthology.orchestra.run.smithi002.stderr: sudo /home/ubuntu/cephtest/cephadm shell --fsid 3aad27a2-7489-11ef-bceb-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-09-17T00:11:53.346 INFO:teuthology.orchestra.run.smithi002.stderr: 2024-09-17T00:11:53.346 INFO:teuthology.orchestra.run.smithi002.stderr:Please consider enabling telemetry to help improve Ceph: 2024-09-17T00:11:53.346 INFO:teuthology.orchestra.run.smithi002.stderr: 2024-09-17T00:11:53.346 INFO:teuthology.orchestra.run.smithi002.stderr: ceph telemetry on 2024-09-17T00:11:53.346 INFO:teuthology.orchestra.run.smithi002.stderr: 2024-09-17T00:11:53.346 INFO:teuthology.orchestra.run.smithi002.stderr:For more information see: 2024-09-17T00:11:53.346 INFO:teuthology.orchestra.run.smithi002.stderr: 2024-09-17T00:11:53.346 INFO:teuthology.orchestra.run.smithi002.stderr: https://docs.ceph.com/docs/master/mgr/telemetry/ 2024-09-17T00:11:53.347 INFO:teuthology.orchestra.run.smithi002.stderr: 2024-09-17T00:11:53.347 INFO:teuthology.orchestra.run.smithi002.stderr:Bootstrap complete. 2024-09-17T00:11:53.384 INFO:tasks.cephadm:Fetching config... 2024-09-17T00:11:53.384 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-09-17T00:11:53.384 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-09-17T00:11:53.402 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-09-17T00:11:53.402 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-09-17T00:11:53.402 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-09-17T00:11:53.458 INFO:tasks.cephadm:Fetching mon keyring... 2024-09-17T00:11:53.458 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-09-17T00:11:53.458 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd if=/var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/mon.a/keyring of=/dev/stdout 2024-09-17T00:11:53.524 INFO:tasks.cephadm:Fetching pub ssh key... 2024-09-17T00:11:53.525 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-09-17T00:11:53.525 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-09-17T00:11:53.580 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-09-17T00:11:53.580 DEBUG:teuthology.orchestra.run.smithi002:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCY/LAtbkiw9PU/IdeC4kdvvnE6A1dntm01o3RQJIeQ2ZiT3I+gDKlS/btW0o7gnB0xAUtQ/jyPRRSE+W14yDF7sSQHIYNce//9gSM5ONgn2JuwYHR2O0UJmKdYuMT4+3PSIGDgJ92MICkdfLYX9pdN1bZnA7fw2g2TWV+birej94tOtvrHDQEG7Q3Hr1E8xyCd/Cnr2Ap3yqKxAoV11EdXPZQvIt7VlTFdGQ4ibpJhdtX2tmj6bi/FDbo34V6SRhfYAJa8BD5dhGzc8i/olx3YNFa9xdii+tHBhsmys1w4uhRpb+gG3B8l9ZIHNVbHHs/xE8AWCFYM45e8Zc+m0TJcKJx7JEMUczq6p3971XNXFPCbJLqOCZG7FDskEBNXFiK0rdJbeXa55m8osFppuN595INtZj/8FqMI3KKcO3iMlcFXLceladxQtSRHLRdJSXkd3bDvJrC7OuyX66Yg5ZRx1mGTHp8YKTI7vULmuIbt+ORlZgyAfzvuSUMjawXYq0s= ceph-3aad27a2-7489-11ef-bceb-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-09-17T00:11:53.658 INFO:teuthology.orchestra.run.smithi002.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCY/LAtbkiw9PU/IdeC4kdvvnE6A1dntm01o3RQJIeQ2ZiT3I+gDKlS/btW0o7gnB0xAUtQ/jyPRRSE+W14yDF7sSQHIYNce//9gSM5ONgn2JuwYHR2O0UJmKdYuMT4+3PSIGDgJ92MICkdfLYX9pdN1bZnA7fw2g2TWV+birej94tOtvrHDQEG7Q3Hr1E8xyCd/Cnr2Ap3yqKxAoV11EdXPZQvIt7VlTFdGQ4ibpJhdtX2tmj6bi/FDbo34V6SRhfYAJa8BD5dhGzc8i/olx3YNFa9xdii+tHBhsmys1w4uhRpb+gG3B8l9ZIHNVbHHs/xE8AWCFYM45e8Zc+m0TJcKJx7JEMUczq6p3971XNXFPCbJLqOCZG7FDskEBNXFiK0rdJbeXa55m8osFppuN595INtZj/8FqMI3KKcO3iMlcFXLceladxQtSRHLRdJSXkd3bDvJrC7OuyX66Yg5ZRx1mGTHp8YKTI7vULmuIbt+ORlZgyAfzvuSUMjawXYq0s= ceph-3aad27a2-7489-11ef-bceb-c7b262605968 2024-09-17T00:11:53.669 DEBUG:teuthology.orchestra.run.smithi099:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCY/LAtbkiw9PU/IdeC4kdvvnE6A1dntm01o3RQJIeQ2ZiT3I+gDKlS/btW0o7gnB0xAUtQ/jyPRRSE+W14yDF7sSQHIYNce//9gSM5ONgn2JuwYHR2O0UJmKdYuMT4+3PSIGDgJ92MICkdfLYX9pdN1bZnA7fw2g2TWV+birej94tOtvrHDQEG7Q3Hr1E8xyCd/Cnr2Ap3yqKxAoV11EdXPZQvIt7VlTFdGQ4ibpJhdtX2tmj6bi/FDbo34V6SRhfYAJa8BD5dhGzc8i/olx3YNFa9xdii+tHBhsmys1w4uhRpb+gG3B8l9ZIHNVbHHs/xE8AWCFYM45e8Zc+m0TJcKJx7JEMUczq6p3971XNXFPCbJLqOCZG7FDskEBNXFiK0rdJbeXa55m8osFppuN595INtZj/8FqMI3KKcO3iMlcFXLceladxQtSRHLRdJSXkd3bDvJrC7OuyX66Yg5ZRx1mGTHp8YKTI7vULmuIbt+ORlZgyAfzvuSUMjawXYq0s= ceph-3aad27a2-7489-11ef-bceb-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-09-17T00:11:53.795 INFO:teuthology.orchestra.run.smithi099.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCY/LAtbkiw9PU/IdeC4kdvvnE6A1dntm01o3RQJIeQ2ZiT3I+gDKlS/btW0o7gnB0xAUtQ/jyPRRSE+W14yDF7sSQHIYNce//9gSM5ONgn2JuwYHR2O0UJmKdYuMT4+3PSIGDgJ92MICkdfLYX9pdN1bZnA7fw2g2TWV+birej94tOtvrHDQEG7Q3Hr1E8xyCd/Cnr2Ap3yqKxAoV11EdXPZQvIt7VlTFdGQ4ibpJhdtX2tmj6bi/FDbo34V6SRhfYAJa8BD5dhGzc8i/olx3YNFa9xdii+tHBhsmys1w4uhRpb+gG3B8l9ZIHNVbHHs/xE8AWCFYM45e8Zc+m0TJcKJx7JEMUczq6p3971XNXFPCbJLqOCZG7FDskEBNXFiK0rdJbeXa55m8osFppuN595INtZj/8FqMI3KKcO3iMlcFXLceladxQtSRHLRdJSXkd3bDvJrC7OuyX66Yg5ZRx1mGTHp8YKTI7vULmuIbt+ORlZgyAfzvuSUMjawXYq0s= ceph-3aad27a2-7489-11ef-bceb-c7b262605968 2024-09-17T00:11:53.808 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3aad27a2-7489-11ef-bceb-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-09-17T00:11:54.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:11:53 smithi002 ceph-mon[23737]: from='client.? 172.21.15.2:0/886569667' entity='client.admin' 2024-09-17T00:11:57.013 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:11:56 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:11:57.013 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:11:56 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi002", "name": "osd_memory_target"}]: dispatch 2024-09-17T00:11:57.014 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:11:56 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:11:57.014 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:11:56 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T00:11:57.014 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:11:56 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:11:57.014 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:11:56 smithi002 ceph-mon[23737]: from='client.? 172.21.15.2:0/1044128421' entity='client.admin' 2024-09-17T00:11:57.014 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:11:56 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T00:11:57.014 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:11:56 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:11:57.039 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-09-17T00:11:57.039 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3aad27a2-7489-11ef-bceb-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-09-17T00:11:59.617 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi099 2024-09-17T00:11:59.617 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-09-17T00:11:59.617 DEBUG:teuthology.orchestra.run.smithi099:> dd of=/etc/ceph/ceph.conf 2024-09-17T00:11:59.637 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-09-17T00:11:59.637 DEBUG:teuthology.orchestra.run.smithi099:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-17T00:11:59.694 INFO:tasks.cephadm:Adding host smithi099 to orchestrator... 2024-09-17T00:11:59.695 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3aad27a2-7489-11ef-bceb-c7b262605968 -- ceph orch host add smithi099 2024-09-17T00:12:00.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:11:59 smithi002 ceph-mon[23737]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:12:00.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:11:59 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:12:00.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:11:59 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T00:12:00.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:11:59 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:12:00.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:11:59 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:12:00.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:11:59 smithi002 ceph-mon[23737]: Updating smithi002:/etc/ceph/ceph.conf 2024-09-17T00:12:00.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:11:59 smithi002 ceph-mon[23737]: Updating smithi002:/etc/ceph/ceph.client.admin.keyring 2024-09-17T00:12:00.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:11:59 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:12:00.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:11:59 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:12:02.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:01 smithi002 ceph-mon[23737]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi099", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:12:02.517 INFO:teuthology.orchestra.run.smithi002.stdout:Added host 'smithi099' with addr '172.21.15.99' 2024-09-17T00:12:03.131 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3aad27a2-7489-11ef-bceb-c7b262605968 -- ceph orch host ls --format=json 2024-09-17T00:12:03.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:03 smithi002 ceph-mon[23737]: Deploying cephadm binary to smithi099 2024-09-17T00:12:03.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:03 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:12:03.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:03 smithi002 ceph-mon[23737]: Added host smithi099 2024-09-17T00:12:03.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:03 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T00:12:03.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:03 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:12:03.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:03 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:12:03.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:03 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:12:03.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:03 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:12:05.085 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-17T00:12:05.085 INFO:teuthology.orchestra.run.smithi002.stdout:[{"addr": "172.21.15.2", "hostname": "smithi002", "labels": [], "status": ""}, {"addr": "172.21.15.99", "hostname": "smithi099", "labels": [], "status": ""}] 2024-09-17T00:12:05.642 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:05 smithi002 ceph-mon[23737]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T00:12:05.642 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:05 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:12:05.732 INFO:tasks.cephadm:Setting crush tunables to default 2024-09-17T00:12:05.732 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3aad27a2-7489-11ef-bceb-c7b262605968 -- ceph osd crush tunables default 2024-09-17T00:12:06.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:06 smithi002 ceph-mon[23737]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T00:12:07.410 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:07 smithi002 ceph-mon[23737]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T00:12:07.410 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:07 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:12:07.410 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:07 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:12:07.410 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:07 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi099", "name": "osd_memory_target"}]: dispatch 2024-09-17T00:12:07.411 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:07 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:12:08.507 INFO:teuthology.orchestra.run.smithi002.stderr:adjusted tunables profile to default 2024-09-17T00:12:08.812 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:08 smithi002 ceph-mon[23737]: from='client.? 172.21.15.2:0/4097472516' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-09-17T00:12:09.091 INFO:tasks.cephadm:Adding mon.a on smithi002 2024-09-17T00:12:09.091 INFO:tasks.cephadm:Adding mon.c on smithi002 2024-09-17T00:12:09.091 INFO:tasks.cephadm:Adding mon.b on smithi099 2024-09-17T00:12:09.091 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3aad27a2-7489-11ef-bceb-c7b262605968 -- ceph orch apply mon '3;smithi002:172.21.15.2=a;smithi002:[v2:172.21.15.2:3301,v1:172.21.15.2:6790]=c;smithi099:172.21.15.99=b' 2024-09-17T00:12:09.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:09 smithi002 ceph-mon[23737]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T00:12:09.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:09 smithi002 ceph-mon[23737]: from='client.? 172.21.15.2:0/4097472516' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-09-17T00:12:09.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:09 smithi002 ceph-mon[23737]: osdmap e4: 0 total, 0 up, 0 in 2024-09-17T00:12:11.051 INFO:teuthology.orchestra.run.smithi099.stdout:Scheduled mon update... 2024-09-17T00:12:11.722 DEBUG:teuthology.orchestra.run.smithi002:mon.c> sudo journalctl -f -n 0 -u ceph-3aad27a2-7489-11ef-bceb-c7b262605968@mon.c.service 2024-09-17T00:12:11.725 DEBUG:teuthology.orchestra.run.smithi099:mon.b> sudo journalctl -f -n 0 -u ceph-3aad27a2-7489-11ef-bceb-c7b262605968@mon.b.service 2024-09-17T00:12:11.727 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-17T00:12:11.727 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3aad27a2-7489-11ef-bceb-c7b262605968 -- ceph mon dump -f json 2024-09-17T00:12:11.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:11 smithi002 ceph-mon[23737]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T00:12:11.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:11 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:12:11.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:11 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T00:12:11.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:11 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:12:11.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:11 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:12:11.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:11 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:12:11.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:11 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:12:11.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:11 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-17T00:12:11.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:11 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:12:12.826 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:12 smithi002 ceph-mon[23737]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi002:172.21.15.2=a;smithi002:[v2:172.21.15.2:3301,v1:172.21.15.2:6790]=c;smithi099:172.21.15.99=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:12:12.827 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:12 smithi002 ceph-mon[23737]: Saving service mon spec with placement smithi002:172.21.15.2=a;smithi002:[v2:172.21.15.2:3301,v1:172.21.15.2:6790]=c;smithi099:172.21.15.99=b;count:3 2024-09-17T00:12:12.827 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:12 smithi002 ceph-mon[23737]: Updating smithi099:/etc/ceph/ceph.conf 2024-09-17T00:12:12.827 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:12 smithi002 ceph-mon[23737]: Updating smithi099:/etc/ceph/ceph.client.admin.keyring 2024-09-17T00:12:12.827 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:12 smithi002 ceph-mon[23737]: Deploying daemon mon.c on smithi002 2024-09-17T00:12:13.777 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-09-17T00:12:13.777 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":1,"fsid":"3aad27a2-7489-11ef-bceb-c7b262605968","modified":"2024-09-17T00:10:42.458448Z","created":"2024-09-17T00:10:42.458448Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:3300","nonce":0},{"type":"v1","addr":"172.21.15.2:6789","nonce":0}]},"addr":"172.21.15.2:6789/0","public_addr":"172.21.15.2:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-17T00:12:13.779 INFO:teuthology.orchestra.run.smithi099.stderr:dumped monmap epoch 1 2024-09-17T00:12:13.792 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:13 smithi002 ceph-mon[23737]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T00:12:14.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 podman[28184]: 2024-09-17 00:12:14.388717927 +0000 UTC m=+0.117429313 container create 77b3d2e1489987946f3f9bd00f85f233eab9bb2c9986788543474445a5661df7 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mon-c, io.k8s.display-name=CentOS Stream 8, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-type=git, ceph=True, RELEASE=HEAD, vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, version=8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, CEPH_POINT_RELEASE=-17.2.0, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.expose-services=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.19.8, release=754, name=centos-stream, distribution-scope=public, io.openshift.tags=base centos centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, architecture=x86_64, build-date=2022-05-03T08:36:31.336870, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_CLEAN=True, com.redhat.component=centos-stream-container, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac) 2024-09-17T00:12:14.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[23737]: from='client.? 172.21.15.99:0/252729467' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-17T00:12:14.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 podman[28184]: 2024-09-17 00:12:14.590970949 +0000 UTC m=+0.319682300 container init 77b3d2e1489987946f3f9bd00f85f233eab9bb2c9986788543474445a5661df7 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mon-c, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, version=8, name=centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-type=git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.tags=base centos centos-stream, maintainer=Guillaume Abrioux , build-date=2022-05-03T08:36:31.336870, release=754, com.redhat.component=centos-stream-container, GIT_REPO=https://github.com/ceph/ceph-container.git, vendor=Red Hat, Inc., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, ceph=True, io.k8s.display-name=CentOS Stream 8, io.buildah.version=1.19.8, architecture=x86_64, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=HEAD, io.openshift.expose-services=, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., CEPH_POINT_RELEASE=-17.2.0, distribution-scope=public, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com) 2024-09-17T00:12:14.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 podman[28184]: 2024-09-17 00:12:14.594857794 +0000 UTC m=+0.323569147 container start 77b3d2e1489987946f3f9bd00f85f233eab9bb2c9986788543474445a5661df7 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mon-c, RELEASE=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, ceph=True, io.openshift.expose-services=, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, build-date=2022-05-03T08:36:31.336870, version=8, vendor=Red Hat, Inc., name=centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base centos centos-stream, vcs-type=git, architecture=x86_64, io.buildah.version=1.19.8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, release=754, GIT_BRANCH=HEAD, com.redhat.component=centos-stream-container, CEPH_POINT_RELEASE=-17.2.0, distribution-scope=public, maintainer=Guillaume Abrioux , io.k8s.display-name=CentOS Stream 8, GIT_REPO=https://github.com/ceph/ceph-container.git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_CLEAN=True) 2024-09-17T00:12:14.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: set uid:gid to 167:167 (ceph:ceph) 2024-09-17T00:12:14.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable), process ceph-mon, pid 2 2024-09-17T00:12:14.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: pidfile_write: ignore empty --pid-file 2024-09-17T00:12:14.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: load: jerasure load: lrc 2024-09-17T00:12:14.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: RocksDB version: 6.15.5 2024-09-17T00:12:14.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-09-17T00:12:14.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Compile date Apr 18 2022 2024-09-17T00:12:14.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: DB SUMMARY 2024-09-17T00:12:14.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: DB Session ID: I6RRA97IX3SGR5OL0ZM7 2024-09-17T00:12:14.981 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: CURRENT file: CURRENT 2024-09-17T00:12:14.981 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: IDENTITY file: IDENTITY 2024-09-17T00:12:14.981 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: MANIFEST file: MANIFEST-000003 size: 57 Bytes 2024-09-17T00:12:14.981 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-09-17T00:12:14.981 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 511 ; 2024-09-17T00:12:14.981 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.error_if_exists: 0 2024-09-17T00:12:14.981 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.create_if_missing: 0 2024-09-17T00:12:14.981 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.paranoid_checks: 1 2024-09-17T00:12:14.981 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-17T00:12:14.981 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.env: 0x5649bd2c6860 2024-09-17T00:12:14.981 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.fs: Posix File System 2024-09-17T00:12:14.981 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.info_log: 0x5649bf943dc0 2024-09-17T00:12:14.981 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.max_file_opening_threads: 16 2024-09-17T00:12:14.981 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.statistics: (nil) 2024-09-17T00:12:14.981 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.use_fsync: 0 2024-09-17T00:12:14.982 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.max_log_file_size: 0 2024-09-17T00:12:14.982 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-17T00:12:14.982 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-17T00:12:14.982 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.keep_log_file_num: 1000 2024-09-17T00:12:14.982 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.recycle_log_file_num: 0 2024-09-17T00:12:14.982 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.allow_fallocate: 1 2024-09-17T00:12:14.982 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.allow_mmap_reads: 0 2024-09-17T00:12:14.982 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.allow_mmap_writes: 0 2024-09-17T00:12:14.982 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.use_direct_reads: 0 2024-09-17T00:12:14.982 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-17T00:12:14.982 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.create_missing_column_families: 0 2024-09-17T00:12:14.982 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.db_log_dir: 2024-09-17T00:12:14.982 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-c/store.db 2024-09-17T00:12:14.983 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-17T00:12:14.983 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-17T00:12:14.983 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-17T00:12:14.983 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-17T00:12:14.983 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-17T00:12:14.983 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-17T00:12:14.984 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.advise_random_on_open: 1 2024-09-17T00:12:14.984 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.db_write_buffer_size: 0 2024-09-17T00:12:14.984 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.write_buffer_manager: 0x5649bfa342d0 2024-09-17T00:12:14.984 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-17T00:12:14.984 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-09-17T00:12:14.984 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-17T00:12:14.985 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-17T00:12:14.985 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.rate_limiter: (nil) 2024-09-17T00:12:14.985 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-17T00:12:14.985 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.wal_recovery_mode: 2 2024-09-17T00:12:14.985 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.enable_thread_tracking: 0 2024-09-17T00:12:14.985 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.enable_pipelined_write: 0 2024-09-17T00:12:14.985 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.unordered_write: 0 2024-09-17T00:12:14.985 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-17T00:12:14.985 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-17T00:12:14.985 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-17T00:12:14.985 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-17T00:12:14.985 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.row_cache: None 2024-09-17T00:12:14.985 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.wal_filter: None 2024-09-17T00:12:14.985 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-17T00:12:14.986 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.allow_ingest_behind: 0 2024-09-17T00:12:14.986 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.preserve_deletes: 0 2024-09-17T00:12:14.986 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.two_write_queues: 0 2024-09-17T00:12:14.986 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.manual_wal_flush: 0 2024-09-17T00:12:14.986 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.atomic_flush: 0 2024-09-17T00:12:14.986 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-17T00:12:14.986 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-17T00:12:14.986 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-17T00:12:14.986 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.log_readahead_size: 0 2024-09-17T00:12:14.986 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-17T00:12:14.986 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.best_efforts_recovery: 0 2024-09-17T00:12:14.986 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-17T00:12:14.986 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-17T00:12:14.986 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.allow_data_in_errors: 0 2024-09-17T00:12:14.987 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.db_host_id: __hostname__ 2024-09-17T00:12:14.987 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.max_background_jobs: 2 2024-09-17T00:12:14.987 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.max_background_compactions: -1 2024-09-17T00:12:14.987 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.max_subcompactions: 1 2024-09-17T00:12:14.987 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-17T00:12:14.987 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-17T00:12:14.987 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-17T00:12:14.987 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.max_total_wal_size: 0 2024-09-17T00:12:14.987 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-17T00:12:14.987 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-17T00:12:14.987 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-17T00:12:14.987 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-17T00:12:14.987 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.max_open_files: -1 2024-09-17T00:12:14.988 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.bytes_per_sync: 0 2024-09-17T00:12:14.988 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-17T00:12:14.988 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-17T00:12:14.988 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.compaction_readahead_size: 0 2024-09-17T00:12:14.988 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.max_background_flushes: -1 2024-09-17T00:12:14.988 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Compression algorithms supported: 2024-09-17T00:12:14.988 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-17T00:12:14.988 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: kZSTD supported: 0 2024-09-17T00:12:14.988 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: kXpressCompression supported: 0 2024-09-17T00:12:14.988 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: kLZ4HCCompression supported: 1 2024-09-17T00:12:14.988 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: kLZ4Compression supported: 1 2024-09-17T00:12:14.988 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: kBZip2Compression supported: 0 2024-09-17T00:12:14.988 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: kZlibCompression supported: 1 2024-09-17T00:12:14.988 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: kSnappyCompression supported: 1 2024-09-17T00:12:14.989 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-17T00:12:14.989 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: [db/version_set.cc:4725] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000003 2024-09-17T00:12:14.989 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: [db/column_family.cc:597] --------------- Options for column family [default]: 2024-09-17T00:12:14.989 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-17T00:12:14.989 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.merge_operator: 2024-09-17T00:12:14.989 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.compaction_filter: None 2024-09-17T00:12:14.989 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.compaction_filter_factory: None 2024-09-17T00:12:14.989 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.sst_partitioner_factory: None 2024-09-17T00:12:14.989 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-17T00:12:14.989 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-17T00:12:14.989 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5649bf90fd00) 2024-09-17T00:12:14.989 INFO:journalctl@ceph.mon.c.smithi002.stdout: cache_index_and_filter_blocks: 1 2024-09-17T00:12:14.990 INFO:journalctl@ceph.mon.c.smithi002.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-17T00:12:14.990 INFO:journalctl@ceph.mon.c.smithi002.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-17T00:12:14.990 INFO:journalctl@ceph.mon.c.smithi002.stdout: pin_top_level_index_and_filter: 1 2024-09-17T00:12:14.990 INFO:journalctl@ceph.mon.c.smithi002.stdout: index_type: 0 2024-09-17T00:12:14.990 INFO:journalctl@ceph.mon.c.smithi002.stdout: data_block_index_type: 0 2024-09-17T00:12:14.990 INFO:journalctl@ceph.mon.c.smithi002.stdout: index_shortening: 1 2024-09-17T00:12:14.990 INFO:journalctl@ceph.mon.c.smithi002.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-17T00:12:14.990 INFO:journalctl@ceph.mon.c.smithi002.stdout: hash_index_allow_collision: 1 2024-09-17T00:12:14.990 INFO:journalctl@ceph.mon.c.smithi002.stdout: checksum: 1 2024-09-17T00:12:14.990 INFO:journalctl@ceph.mon.c.smithi002.stdout: no_block_cache: 0 2024-09-17T00:12:14.990 INFO:journalctl@ceph.mon.c.smithi002.stdout: block_cache: 0x5649bf97a170 2024-09-17T00:12:14.990 INFO:journalctl@ceph.mon.c.smithi002.stdout: block_cache_name: BinnedLRUCache 2024-09-17T00:12:14.990 INFO:journalctl@ceph.mon.c.smithi002.stdout: block_cache_options: 2024-09-17T00:12:14.990 INFO:journalctl@ceph.mon.c.smithi002.stdout: capacity : 536870912 2024-09-17T00:12:14.991 INFO:journalctl@ceph.mon.c.smithi002.stdout: num_shard_bits : 4 2024-09-17T00:12:14.991 INFO:journalctl@ceph.mon.c.smithi002.stdout: strict_capacity_limit : 0 2024-09-17T00:12:14.991 INFO:journalctl@ceph.mon.c.smithi002.stdout: high_pri_pool_ratio: 0.000 2024-09-17T00:12:14.991 INFO:journalctl@ceph.mon.c.smithi002.stdout: block_cache_compressed: (nil) 2024-09-17T00:12:14.991 INFO:journalctl@ceph.mon.c.smithi002.stdout: persistent_cache: (nil) 2024-09-17T00:12:14.991 INFO:journalctl@ceph.mon.c.smithi002.stdout: block_size: 4096 2024-09-17T00:12:14.991 INFO:journalctl@ceph.mon.c.smithi002.stdout: block_size_deviation: 10 2024-09-17T00:12:14.991 INFO:journalctl@ceph.mon.c.smithi002.stdout: block_restart_interval: 16 2024-09-17T00:12:14.991 INFO:journalctl@ceph.mon.c.smithi002.stdout: index_block_restart_interval: 1 2024-09-17T00:12:14.991 INFO:journalctl@ceph.mon.c.smithi002.stdout: metadata_block_size: 4096 2024-09-17T00:12:14.991 INFO:journalctl@ceph.mon.c.smithi002.stdout: partition_filters: 0 2024-09-17T00:12:14.991 INFO:journalctl@ceph.mon.c.smithi002.stdout: use_delta_encoding: 1 2024-09-17T00:12:14.991 INFO:journalctl@ceph.mon.c.smithi002.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-09-17T00:12:14.991 INFO:journalctl@ceph.mon.c.smithi002.stdout: whole_key_filtering: 1 2024-09-17T00:12:14.992 INFO:journalctl@ceph.mon.c.smithi002.stdout: verify_compression: 0 2024-09-17T00:12:14.992 INFO:journalctl@ceph.mon.c.smithi002.stdout: read_amp_bytes_per_bit: 0 2024-09-17T00:12:14.992 INFO:journalctl@ceph.mon.c.smithi002.stdout: format_version: 4 2024-09-17T00:12:14.992 INFO:journalctl@ceph.mon.c.smithi002.stdout: enable_index_compression: 1 2024-09-17T00:12:14.992 INFO:journalctl@ceph.mon.c.smithi002.stdout: block_align: 0 2024-09-17T00:12:14.992 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.write_buffer_size: 33554432 2024-09-17T00:12:14.992 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.max_write_buffer_number: 2 2024-09-17T00:12:14.992 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.compression: NoCompression 2024-09-17T00:12:14.992 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.bottommost_compression: Disabled 2024-09-17T00:12:14.992 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.prefix_extractor: nullptr 2024-09-17T00:12:14.992 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-17T00:12:14.992 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.num_levels: 7 2024-09-17T00:12:14.992 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-17T00:12:14.992 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-17T00:12:14.993 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-17T00:12:14.993 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-17T00:12:14.993 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-17T00:12:14.993 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-17T00:12:14.993 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-17T00:12:14.993 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-17T00:12:14.993 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-17T00:12:14.993 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-17T00:12:14.993 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-17T00:12:14.993 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.compression_opts.level: 32767 2024-09-17T00:12:14.993 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.compression_opts.strategy: 0 2024-09-17T00:12:14.993 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-17T00:12:14.993 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-17T00:12:14.994 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-17T00:12:14.994 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.compression_opts.enabled: false 2024-09-17T00:12:14.994 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-17T00:12:14.994 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-17T00:12:14.994 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-17T00:12:14.994 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.target_file_size_base: 67108864 2024-09-17T00:12:14.994 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-17T00:12:14.994 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-17T00:12:14.994 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-17T00:12:14.994 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-17T00:12:14.994 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-17T00:12:14.994 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-17T00:12:14.994 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-17T00:12:14.994 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-17T00:12:14.995 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-17T00:12:14.995 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-17T00:12:14.995 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-17T00:12:14.995 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-17T00:12:14.995 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-17T00:12:14.995 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.arena_block_size: 4194304 2024-09-17T00:12:14.995 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-17T00:12:14.995 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-17T00:12:14.995 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-09-17T00:12:14.995 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.disable_auto_compactions: 0 2024-09-17T00:12:14.995 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-17T00:12:14.995 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-17T00:12:14.995 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-17T00:12:14.995 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-17T00:12:14.996 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-17T00:12:14.996 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-17T00:12:14.996 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-17T00:12:14.996 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-17T00:12:14.996 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-17T00:12:14.996 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-17T00:12:14.996 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.table_properties_collectors: 2024-09-17T00:12:14.996 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.inplace_update_support: 0 2024-09-17T00:12:14.996 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-17T00:12:14.996 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-17T00:12:14.996 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-17T00:12:14.996 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-17T00:12:14.996 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.bloom_locality: 0 2024-09-17T00:12:14.997 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.max_successive_merges: 0 2024-09-17T00:12:14.997 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-17T00:12:14.997 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.paranoid_file_checks: 0 2024-09-17T00:12:14.997 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.force_consistency_checks: 1 2024-09-17T00:12:14.997 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.report_bg_io_stats: 0 2024-09-17T00:12:14.997 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.ttl: 2592000 2024-09-17T00:12:14.997 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-17T00:12:14.997 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.enable_blob_files: false 2024-09-17T00:12:14.997 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.min_blob_size: 0 2024-09-17T00:12:14.997 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.blob_file_size: 268435456 2024-09-17T00:12:14.997 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-17T00:12:14.997 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-17T00:12:14.997 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-17T00:12:14.997 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: [db/version_set.cc:4773] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000003 succeeded,manifest_file_number is 3, next_file_number is 5, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-09-17T00:12:14.998 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: [db/version_set.cc:4782] Column family [default] (ID 0), log number is 0 2024-09-17T00:12:14.998 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: [db/version_set.cc:4083] Creating manifest 7 2024-09-17T00:12:14.998 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726531934621733, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-09-17T00:12:14.998 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: [db/db_impl/db_impl_open.cc:847] Recovering log #4 mode 2 2024-09-17T00:12:14.998 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: [table/block_based/filter_policy.cc:996] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-09-17T00:12:14.998 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726531934622058, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1545, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1726531934, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "0ce868a4-e1c2-49ed-a167-f4ec8e7d5d34", "db_session_id": "I6RRA97IX3SGR5OL0ZM7"}} 2024-09-17T00:12:14.998 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: [db/version_set.cc:4083] Creating manifest 9 2024-09-17T00:12:14.998 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726531934622378, "job": 1, "event": "recovery_finished"} 2024-09-17T00:12:14.998 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: [file/delete_scheduler.cc:73] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-17T00:12:14.998 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: [db/db_impl/db_impl_open.cc:1701] SstFileManager instance 0x5649bf960a80 2024-09-17T00:12:14.998 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: DB pointer 0x5649bf970000 2024-09-17T00:12:14.998 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: [db/db_impl/db_impl.cc:902] ------- DUMPING STATS ------- 2024-09-17T00:12:14.998 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-09-17T00:12:14.998 INFO:journalctl@ceph.mon.c.smithi002.stdout: ** DB Stats ** 2024-09-17T00:12:14.999 INFO:journalctl@ceph.mon.c.smithi002.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-17T00:12:14.999 INFO:journalctl@ceph.mon.c.smithi002.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-17T00:12:14.999 INFO:journalctl@ceph.mon.c.smithi002.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-17T00:12:14.999 INFO:journalctl@ceph.mon.c.smithi002.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-17T00:12:14.999 INFO:journalctl@ceph.mon.c.smithi002.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-17T00:12:14.999 INFO:journalctl@ceph.mon.c.smithi002.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-09-17T00:12:14.999 INFO:journalctl@ceph.mon.c.smithi002.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-17T00:12:14.999 INFO:journalctl@ceph.mon.c.smithi002.stdout: 2024-09-17T00:12:14.999 INFO:journalctl@ceph.mon.c.smithi002.stdout: ** Compaction Stats [default] ** 2024-09-17T00:12:14.999 INFO:journalctl@ceph.mon.c.smithi002.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-17T00:12:14.999 INFO:journalctl@ceph.mon.c.smithi002.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-17T00:12:14.999 INFO:journalctl@ceph.mon.c.smithi002.stdout: L0 1/0 1.51 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.8 0.00 0.00 1 0.000 0 0 2024-09-17T00:12:14.999 INFO:journalctl@ceph.mon.c.smithi002.stdout: Sum 1/0 1.51 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.8 0.00 0.00 1 0.000 0 0 2024-09-17T00:12:14.999 INFO:journalctl@ceph.mon.c.smithi002.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.8 0.00 0.00 1 0.000 0 0 2024-09-17T00:12:15.000 INFO:journalctl@ceph.mon.c.smithi002.stdout: 2024-09-17T00:12:15.000 INFO:journalctl@ceph.mon.c.smithi002.stdout: ** Compaction Stats [default] ** 2024-09-17T00:12:15.000 INFO:journalctl@ceph.mon.c.smithi002.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-17T00:12:15.000 INFO:journalctl@ceph.mon.c.smithi002.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-17T00:12:15.000 INFO:journalctl@ceph.mon.c.smithi002.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.8 0.00 0.00 1 0.000 0 0 2024-09-17T00:12:15.000 INFO:journalctl@ceph.mon.c.smithi002.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-17T00:12:15.000 INFO:journalctl@ceph.mon.c.smithi002.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-17T00:12:15.000 INFO:journalctl@ceph.mon.c.smithi002.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-17T00:12:15.000 INFO:journalctl@ceph.mon.c.smithi002.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-17T00:12:15.000 INFO:journalctl@ceph.mon.c.smithi002.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-17T00:12:15.000 INFO:journalctl@ceph.mon.c.smithi002.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-17T00:12:15.000 INFO:journalctl@ceph.mon.c.smithi002.stdout: Cumulative compaction: 0.00 GB write, 0.56 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-17T00:12:15.000 INFO:journalctl@ceph.mon.c.smithi002.stdout: Interval compaction: 0.00 GB write, 0.56 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-17T00:12:15.000 INFO:journalctl@ceph.mon.c.smithi002.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-17T00:12:15.000 INFO:journalctl@ceph.mon.c.smithi002.stdout: 2024-09-17T00:12:15.001 INFO:journalctl@ceph.mon.c.smithi002.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-17T00:12:15.001 INFO:journalctl@ceph.mon.c.smithi002.stdout: 2024-09-17T00:12:15.001 INFO:journalctl@ceph.mon.c.smithi002.stdout: ** Compaction Stats [default] ** 2024-09-17T00:12:15.001 INFO:journalctl@ceph.mon.c.smithi002.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-17T00:12:15.001 INFO:journalctl@ceph.mon.c.smithi002.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-17T00:12:15.001 INFO:journalctl@ceph.mon.c.smithi002.stdout: L0 1/0 1.51 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.8 0.00 0.00 1 0.000 0 0 2024-09-17T00:12:15.001 INFO:journalctl@ceph.mon.c.smithi002.stdout: Sum 1/0 1.51 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.8 0.00 0.00 1 0.000 0 0 2024-09-17T00:12:15.001 INFO:journalctl@ceph.mon.c.smithi002.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-09-17T00:12:15.001 INFO:journalctl@ceph.mon.c.smithi002.stdout: 2024-09-17T00:12:15.001 INFO:journalctl@ceph.mon.c.smithi002.stdout: ** Compaction Stats [default] ** 2024-09-17T00:12:15.001 INFO:journalctl@ceph.mon.c.smithi002.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-17T00:12:15.001 INFO:journalctl@ceph.mon.c.smithi002.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-17T00:12:15.001 INFO:journalctl@ceph.mon.c.smithi002.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.8 0.00 0.00 1 0.000 0 0 2024-09-17T00:12:15.001 INFO:journalctl@ceph.mon.c.smithi002.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-17T00:12:15.002 INFO:journalctl@ceph.mon.c.smithi002.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-17T00:12:15.002 INFO:journalctl@ceph.mon.c.smithi002.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-17T00:12:15.002 INFO:journalctl@ceph.mon.c.smithi002.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-17T00:12:15.002 INFO:journalctl@ceph.mon.c.smithi002.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-17T00:12:15.002 INFO:journalctl@ceph.mon.c.smithi002.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-17T00:12:15.002 INFO:journalctl@ceph.mon.c.smithi002.stdout: Cumulative compaction: 0.00 GB write, 0.55 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-17T00:12:15.002 INFO:journalctl@ceph.mon.c.smithi002.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-17T00:12:15.002 INFO:journalctl@ceph.mon.c.smithi002.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-17T00:12:15.002 INFO:journalctl@ceph.mon.c.smithi002.stdout: 2024-09-17T00:12:15.002 INFO:journalctl@ceph.mon.c.smithi002.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-17T00:12:15.002 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-09-17T00:12:15.002 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: using public_addrv [v2:172.21.15.2:3301/0,v1:172.21.15.2:6790/0] 2024-09-17T00:12:15.003 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: starting mon.c rank -1 at public addrs [v2:172.21.15.2:3301/0,v1:172.21.15.2:6790/0] at bind addrs [v2:172.21.15.2:3301/0,v1:172.21.15.2:6790/0] mon_data /var/lib/ceph/mon/ceph-c fsid 3aad27a2-7489-11ef-bceb-c7b262605968 2024-09-17T00:12:15.003 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: mon.c@-1(???) e0 preinit fsid 3aad27a2-7489-11ef-bceb-c7b262605968 2024-09-17T00:12:15.003 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: mon.c@-1(synchronizing).mds e1 new map 2024-09-17T00:12:15.003 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: mon.c@-1(synchronizing).mds e1 print_map 2024-09-17T00:12:15.003 INFO:journalctl@ceph.mon.c.smithi002.stdout: e1 2024-09-17T00:12:15.003 INFO:journalctl@ceph.mon.c.smithi002.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-17T00:12:15.003 INFO:journalctl@ceph.mon.c.smithi002.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-09-17T00:12:15.003 INFO:journalctl@ceph.mon.c.smithi002.stdout: legacy client fscid: -1 2024-09-17T00:12:15.003 INFO:journalctl@ceph.mon.c.smithi002.stdout: 2024-09-17T00:12:15.003 INFO:journalctl@ceph.mon.c.smithi002.stdout: No filesystems configured 2024-09-17T00:12:15.003 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-09-17T00:12:15.004 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-09-17T00:12:15.004 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-09-17T00:12:15.004 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-09-17T00:12:15.004 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-09-17T00:12:15.004 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-09-17T00:12:15.004 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-09-17T00:12:15.004 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-17T00:12:15.004 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-17T00:12:15.004 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-17T00:12:15.004 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi002:172.21.15.2=a;smithi002:[v2:172.21.15.2:3301,v1:172.21.15.2:6790]=c;smithi099:172.21.15.99=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:12:15.004 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: Saving service mon spec with placement smithi002:172.21.15.2=a;smithi002:[v2:172.21.15.2:3301,v1:172.21.15.2:6790]=c;smithi099:172.21.15.99=b;count:3 2024-09-17T00:12:15.004 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: Updating smithi099:/etc/ceph/ceph.conf 2024-09-17T00:12:15.004 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: Updating smithi099:/etc/ceph/ceph.client.admin.keyring 2024-09-17T00:12:15.004 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: Deploying daemon mon.c on smithi002 2024-09-17T00:12:15.005 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T00:12:15.005 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: from='client.? 172.21.15.99:0/252729467' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-17T00:12:15.005 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-09-17T00:12:15.005 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 ceph-mon[28203]: mon.c@-1(synchronizing) e1 handle_conf_change mon_allow_pool_delete,mon_cluster_log_to_file 2024-09-17T00:12:15.005 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 bash[28184]: 77b3d2e1489987946f3f9bd00f85f233eab9bb2c9986788543474445a5661df7 2024-09-17T00:12:15.005 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:14 smithi002 systemd[1]: Started Ceph mon.c for 3aad27a2-7489-11ef-bceb-c7b262605968. 2024-09-17T00:12:15.343 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-17T00:12:15.343 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3aad27a2-7489-11ef-bceb-c7b262605968 -- ceph mon dump -f json 2024-09-17T00:12:19.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:19 smithi002 ceph-mon[23737]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T00:12:19.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:19 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T00:12:19.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:19 smithi002 ceph-mon[23737]: mon.a calling monitor election 2024-09-17T00:12:19.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:19 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T00:12:19.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:19 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T00:12:19.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:19 smithi002 ceph-mon[23737]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T00:12:19.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:19 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T00:12:19.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:19 smithi002 ceph-mon[23737]: mon.c calling monitor election 2024-09-17T00:12:19.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:19 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T00:12:19.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:19 smithi002 ceph-mon[23737]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T00:12:19.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:19 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T00:12:19.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:19 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T00:12:19.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:19 smithi002 ceph-mon[23737]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-09-17T00:12:19.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:19 smithi002 ceph-mon[23737]: monmap e2: 2 mons at {a=[v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0],c=[v2:172.21.15.2:3301/0,v1:172.21.15.2:6790/0]} 2024-09-17T00:12:19.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:19 smithi002 ceph-mon[23737]: fsmap 2024-09-17T00:12:19.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:19 smithi002 ceph-mon[23737]: osdmap e4: 0 total, 0 up, 0 in 2024-09-17T00:12:19.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:19 smithi002 ceph-mon[23737]: mgrmap e13: y(active, since 35s) 2024-09-17T00:12:19.981 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:19 smithi002 ceph-mon[23737]: overall HEALTH_OK 2024-09-17T00:12:19.981 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:19 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:12:19.981 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:19 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-17T00:12:19.981 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:19 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:12:19.981 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:19 smithi002 ceph-mon[28203]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T00:12:19.981 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:19 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T00:12:19.981 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:19 smithi002 ceph-mon[28203]: mon.a calling monitor election 2024-09-17T00:12:19.981 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:19 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T00:12:19.981 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:19 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T00:12:19.981 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:19 smithi002 ceph-mon[28203]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T00:12:19.981 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:19 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T00:12:19.981 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:19 smithi002 ceph-mon[28203]: mon.c calling monitor election 2024-09-17T00:12:19.982 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:19 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T00:12:19.982 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:19 smithi002 ceph-mon[28203]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T00:12:19.982 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:19 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T00:12:19.982 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:19 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T00:12:19.982 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:19 smithi002 ceph-mon[28203]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-09-17T00:12:19.982 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:19 smithi002 ceph-mon[28203]: monmap e2: 2 mons at {a=[v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0],c=[v2:172.21.15.2:3301/0,v1:172.21.15.2:6790/0]} 2024-09-17T00:12:19.982 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:19 smithi002 ceph-mon[28203]: fsmap 2024-09-17T00:12:19.982 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:19 smithi002 ceph-mon[28203]: osdmap e4: 0 total, 0 up, 0 in 2024-09-17T00:12:19.982 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:19 smithi002 ceph-mon[28203]: mgrmap e13: y(active, since 35s) 2024-09-17T00:12:19.982 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:19 smithi002 ceph-mon[28203]: overall HEALTH_OK 2024-09-17T00:12:19.982 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:19 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:12:19.982 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:19 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-17T00:12:19.982 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:19 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:12:20.217 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-09-17T00:12:20.217 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":2,"fsid":"3aad27a2-7489-11ef-bceb-c7b262605968","modified":"2024-09-17T00:12:14.641462Z","created":"2024-09-17T00:10:42.458448Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:3300","nonce":0},{"type":"v1","addr":"172.21.15.2:6789","nonce":0}]},"addr":"172.21.15.2:6789/0","public_addr":"172.21.15.2:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:3301","nonce":0},{"type":"v1","addr":"172.21.15.2:6790","nonce":0}]},"addr":"172.21.15.2:6790/0","public_addr":"172.21.15.2:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-09-17T00:12:20.219 INFO:teuthology.orchestra.run.smithi099.stderr:dumped monmap epoch 2 2024-09-17T00:12:20.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:20 smithi002 ceph-mon[23737]: Deploying daemon mon.b on smithi099 2024-09-17T00:12:20.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:20 smithi002 ceph-mon[23737]: from='client.? 172.21.15.99:0/4165018019' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-17T00:12:20.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:20 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T00:12:20.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:20 smithi002 ceph-mon[28203]: Deploying daemon mon.b on smithi099 2024-09-17T00:12:20.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:20 smithi002 ceph-mon[28203]: from='client.? 172.21.15.99:0/4165018019' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-17T00:12:20.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:20 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T00:12:21.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:21 smithi002 ceph-mon[28203]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T00:12:21.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:21 smithi002 ceph-mon[23737]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T00:12:22.113 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-17T00:12:22.113 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3aad27a2-7489-11ef-bceb-c7b262605968 -- ceph mon dump -f json 2024-09-17T00:12:23.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:23 smithi099 podman[24539]: 2024-09-17 00:12:23.798315 +0000 UTC m=+0.021839022 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a 2024-09-17T00:12:23.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:23 smithi002 ceph-mon[28203]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T00:12:23.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:23 smithi002 ceph-mon[23737]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T00:12:24.241 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-09-17T00:12:24.242 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":2,"fsid":"3aad27a2-7489-11ef-bceb-c7b262605968","modified":"2024-09-17T00:12:14.641462Z","created":"2024-09-17T00:10:42.458448Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:3300","nonce":0},{"type":"v1","addr":"172.21.15.2:6789","nonce":0}]},"addr":"172.21.15.2:6789/0","public_addr":"172.21.15.2:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:3301","nonce":0},{"type":"v1","addr":"172.21.15.2:6790","nonce":0}]},"addr":"172.21.15.2:6790/0","public_addr":"172.21.15.2:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-09-17T00:12:24.243 INFO:teuthology.orchestra.run.smithi099.stderr:dumped monmap epoch 2 2024-09-17T00:12:24.244 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:23 smithi099 podman[24539]: 2024-09-17 00:12:23.997304206 +0000 UTC m=+0.220828228 container create c6d2af7151872ee22f65b0ab7d833e966a67e7443f3369aa2a7ba9d93c7e9320 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mon-b, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-17.2.0, RELEASE=HEAD, name=centos-stream, io.openshift.tags=base centos centos-stream, release=754, vendor=Red Hat, Inc., io.k8s.display-name=CentOS Stream 8, GIT_CLEAN=True, version=8, distribution-scope=public, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-type=git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.buildah.version=1.19.8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-05-03T08:36:31.336870, architecture=x86_64, maintainer=Guillaume Abrioux , GIT_REPO=https://github.com/ceph/ceph-container.git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.component=centos-stream-container, ceph=True, io.openshift.expose-services=) 2024-09-17T00:12:24.718 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 podman[24539]: 2024-09-17 00:12:24.333660293 +0000 UTC m=+0.557184315 container init c6d2af7151872ee22f65b0ab7d833e966a67e7443f3369aa2a7ba9d93c7e9320 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mon-b, GIT_BRANCH=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, release=754, distribution-scope=public, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., version=8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, name=centos-stream, vcs-type=git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, CEPH_POINT_RELEASE=-17.2.0, vendor=Red Hat, Inc., io.buildah.version=1.19.8, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, build-date=2022-05-03T08:36:31.336870, architecture=x86_64, io.openshift.tags=base centos centos-stream, GIT_CLEAN=True, maintainer=Guillaume Abrioux , com.redhat.component=centos-stream-container, ceph=True, io.openshift.expose-services=, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.display-name=CentOS Stream 8) 2024-09-17T00:12:24.718 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 podman[24539]: 2024-09-17 00:12:24.337306136 +0000 UTC m=+0.560830159 container start c6d2af7151872ee22f65b0ab7d833e966a67e7443f3369aa2a7ba9d93c7e9320 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mon-b, GIT_BRANCH=HEAD, io.k8s.display-name=CentOS Stream 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_REPO=https://github.com/ceph/ceph-container.git, name=centos-stream, io.openshift.tags=base centos centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_CLEAN=True, version=8, io.buildah.version=1.19.8, ceph=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, release=754, maintainer=Guillaume Abrioux , io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-05-03T08:36:31.336870, architecture=x86_64, com.redhat.component=centos-stream-container, vendor=Red Hat, Inc., vcs-type=git, distribution-scope=public, CEPH_POINT_RELEASE=-17.2.0, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.expose-services=, RELEASE=HEAD) 2024-09-17T00:12:24.718 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: set uid:gid to 167:167 (ceph:ceph) 2024-09-17T00:12:24.718 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable), process ceph-mon, pid 2 2024-09-17T00:12:24.718 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: pidfile_write: ignore empty --pid-file 2024-09-17T00:12:24.718 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: load: jerasure load: lrc 2024-09-17T00:12:24.718 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: RocksDB version: 6.15.5 2024-09-17T00:12:24.718 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-09-17T00:12:24.718 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Compile date Apr 18 2022 2024-09-17T00:12:24.718 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: DB SUMMARY 2024-09-17T00:12:24.718 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: DB Session ID: OGVCPTSU3CQKCMZZA07P 2024-09-17T00:12:24.719 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: CURRENT file: CURRENT 2024-09-17T00:12:24.719 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: IDENTITY file: IDENTITY 2024-09-17T00:12:24.719 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: MANIFEST file: MANIFEST-000003 size: 57 Bytes 2024-09-17T00:12:24.719 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2024-09-17T00:12:24.719 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 636 ; 2024-09-17T00:12:24.719 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.error_if_exists: 0 2024-09-17T00:12:24.719 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.create_if_missing: 0 2024-09-17T00:12:24.719 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.paranoid_checks: 1 2024-09-17T00:12:24.719 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-17T00:12:24.719 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.env: 0x559a35ec6860 2024-09-17T00:12:24.719 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.fs: Posix File System 2024-09-17T00:12:24.719 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.info_log: 0x559a383f1dc0 2024-09-17T00:12:24.719 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.max_file_opening_threads: 16 2024-09-17T00:12:24.719 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.statistics: (nil) 2024-09-17T00:12:24.720 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.use_fsync: 0 2024-09-17T00:12:24.720 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.max_log_file_size: 0 2024-09-17T00:12:24.720 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-17T00:12:24.720 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-17T00:12:24.720 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.keep_log_file_num: 1000 2024-09-17T00:12:24.720 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.recycle_log_file_num: 0 2024-09-17T00:12:24.720 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.allow_fallocate: 1 2024-09-17T00:12:24.720 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.allow_mmap_reads: 0 2024-09-17T00:12:24.720 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.allow_mmap_writes: 0 2024-09-17T00:12:24.720 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.use_direct_reads: 0 2024-09-17T00:12:24.720 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-17T00:12:24.720 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.create_missing_column_families: 0 2024-09-17T00:12:24.720 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.db_log_dir: 2024-09-17T00:12:24.720 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-b/store.db 2024-09-17T00:12:24.721 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-17T00:12:24.721 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-17T00:12:24.721 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-17T00:12:24.721 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-17T00:12:24.721 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-17T00:12:24.721 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-17T00:12:24.722 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.advise_random_on_open: 1 2024-09-17T00:12:24.722 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.db_write_buffer_size: 0 2024-09-17T00:12:24.722 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.write_buffer_manager: 0x559a384e2240 2024-09-17T00:12:24.722 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-17T00:12:24.723 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-09-17T00:12:24.723 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-17T00:12:24.723 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-17T00:12:24.723 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.rate_limiter: (nil) 2024-09-17T00:12:24.723 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-17T00:12:24.723 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.wal_recovery_mode: 2 2024-09-17T00:12:24.723 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.enable_thread_tracking: 0 2024-09-17T00:12:24.723 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.enable_pipelined_write: 0 2024-09-17T00:12:24.723 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.unordered_write: 0 2024-09-17T00:12:24.723 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-17T00:12:24.723 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-17T00:12:24.723 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-17T00:12:24.723 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-17T00:12:24.723 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.row_cache: None 2024-09-17T00:12:24.724 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.wal_filter: None 2024-09-17T00:12:24.724 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-17T00:12:24.724 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.allow_ingest_behind: 0 2024-09-17T00:12:24.724 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.preserve_deletes: 0 2024-09-17T00:12:24.724 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.two_write_queues: 0 2024-09-17T00:12:24.724 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.manual_wal_flush: 0 2024-09-17T00:12:24.724 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.atomic_flush: 0 2024-09-17T00:12:24.724 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-17T00:12:24.724 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-17T00:12:24.724 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-17T00:12:24.724 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.log_readahead_size: 0 2024-09-17T00:12:24.724 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-17T00:12:24.724 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.best_efforts_recovery: 0 2024-09-17T00:12:24.724 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-17T00:12:24.724 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-17T00:12:24.725 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.allow_data_in_errors: 0 2024-09-17T00:12:24.725 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.db_host_id: __hostname__ 2024-09-17T00:12:24.725 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.max_background_jobs: 2 2024-09-17T00:12:24.725 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.max_background_compactions: -1 2024-09-17T00:12:24.725 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.max_subcompactions: 1 2024-09-17T00:12:24.725 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-17T00:12:24.725 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-17T00:12:24.725 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-17T00:12:24.725 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.max_total_wal_size: 0 2024-09-17T00:12:24.725 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-17T00:12:24.725 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-17T00:12:24.725 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-17T00:12:24.725 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-17T00:12:24.725 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.max_open_files: -1 2024-09-17T00:12:24.725 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.bytes_per_sync: 0 2024-09-17T00:12:24.726 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-17T00:12:24.726 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-17T00:12:24.726 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.compaction_readahead_size: 0 2024-09-17T00:12:24.726 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.max_background_flushes: -1 2024-09-17T00:12:24.726 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Compression algorithms supported: 2024-09-17T00:12:24.726 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-17T00:12:24.726 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: kZSTD supported: 0 2024-09-17T00:12:24.726 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: kXpressCompression supported: 0 2024-09-17T00:12:24.726 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: kLZ4HCCompression supported: 1 2024-09-17T00:12:24.726 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: kLZ4Compression supported: 1 2024-09-17T00:12:24.726 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: kBZip2Compression supported: 0 2024-09-17T00:12:24.726 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: kZlibCompression supported: 1 2024-09-17T00:12:24.726 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: kSnappyCompression supported: 1 2024-09-17T00:12:24.726 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-17T00:12:24.726 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: [db/version_set.cc:4725] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000003 2024-09-17T00:12:24.727 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: [db/column_family.cc:597] --------------- Options for column family [default]: 2024-09-17T00:12:24.727 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-17T00:12:24.727 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.merge_operator: 2024-09-17T00:12:24.727 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.compaction_filter: None 2024-09-17T00:12:24.727 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.compaction_filter_factory: None 2024-09-17T00:12:24.727 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.sst_partitioner_factory: None 2024-09-17T00:12:24.727 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-17T00:12:24.727 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-17T00:12:24.727 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x559a383bdd00) 2024-09-17T00:12:24.727 INFO:journalctl@ceph.mon.b.smithi099.stdout: cache_index_and_filter_blocks: 1 2024-09-17T00:12:24.727 INFO:journalctl@ceph.mon.b.smithi099.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-17T00:12:24.727 INFO:journalctl@ceph.mon.b.smithi099.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-17T00:12:24.727 INFO:journalctl@ceph.mon.b.smithi099.stdout: pin_top_level_index_and_filter: 1 2024-09-17T00:12:24.727 INFO:journalctl@ceph.mon.b.smithi099.stdout: index_type: 0 2024-09-17T00:12:24.727 INFO:journalctl@ceph.mon.b.smithi099.stdout: data_block_index_type: 0 2024-09-17T00:12:24.728 INFO:journalctl@ceph.mon.b.smithi099.stdout: index_shortening: 1 2024-09-17T00:12:24.728 INFO:journalctl@ceph.mon.b.smithi099.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-17T00:12:24.728 INFO:journalctl@ceph.mon.b.smithi099.stdout: hash_index_allow_collision: 1 2024-09-17T00:12:24.728 INFO:journalctl@ceph.mon.b.smithi099.stdout: checksum: 1 2024-09-17T00:12:24.728 INFO:journalctl@ceph.mon.b.smithi099.stdout: no_block_cache: 0 2024-09-17T00:12:24.728 INFO:journalctl@ceph.mon.b.smithi099.stdout: block_cache: 0x559a38428170 2024-09-17T00:12:24.728 INFO:journalctl@ceph.mon.b.smithi099.stdout: block_cache_name: BinnedLRUCache 2024-09-17T00:12:24.728 INFO:journalctl@ceph.mon.b.smithi099.stdout: block_cache_options: 2024-09-17T00:12:24.728 INFO:journalctl@ceph.mon.b.smithi099.stdout: capacity : 536870912 2024-09-17T00:12:24.728 INFO:journalctl@ceph.mon.b.smithi099.stdout: num_shard_bits : 4 2024-09-17T00:12:24.728 INFO:journalctl@ceph.mon.b.smithi099.stdout: strict_capacity_limit : 0 2024-09-17T00:12:24.728 INFO:journalctl@ceph.mon.b.smithi099.stdout: high_pri_pool_ratio: 0.000 2024-09-17T00:12:24.728 INFO:journalctl@ceph.mon.b.smithi099.stdout: block_cache_compressed: (nil) 2024-09-17T00:12:24.728 INFO:journalctl@ceph.mon.b.smithi099.stdout: persistent_cache: (nil) 2024-09-17T00:12:24.728 INFO:journalctl@ceph.mon.b.smithi099.stdout: block_size: 4096 2024-09-17T00:12:24.729 INFO:journalctl@ceph.mon.b.smithi099.stdout: block_size_deviation: 10 2024-09-17T00:12:24.729 INFO:journalctl@ceph.mon.b.smithi099.stdout: block_restart_interval: 16 2024-09-17T00:12:24.729 INFO:journalctl@ceph.mon.b.smithi099.stdout: index_block_restart_interval: 1 2024-09-17T00:12:24.729 INFO:journalctl@ceph.mon.b.smithi099.stdout: metadata_block_size: 4096 2024-09-17T00:12:24.729 INFO:journalctl@ceph.mon.b.smithi099.stdout: partition_filters: 0 2024-09-17T00:12:24.729 INFO:journalctl@ceph.mon.b.smithi099.stdout: use_delta_encoding: 1 2024-09-17T00:12:24.729 INFO:journalctl@ceph.mon.b.smithi099.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-09-17T00:12:24.729 INFO:journalctl@ceph.mon.b.smithi099.stdout: whole_key_filtering: 1 2024-09-17T00:12:24.729 INFO:journalctl@ceph.mon.b.smithi099.stdout: verify_compression: 0 2024-09-17T00:12:24.729 INFO:journalctl@ceph.mon.b.smithi099.stdout: read_amp_bytes_per_bit: 0 2024-09-17T00:12:24.729 INFO:journalctl@ceph.mon.b.smithi099.stdout: format_version: 4 2024-09-17T00:12:24.729 INFO:journalctl@ceph.mon.b.smithi099.stdout: enable_index_compression: 1 2024-09-17T00:12:24.729 INFO:journalctl@ceph.mon.b.smithi099.stdout: block_align: 0 2024-09-17T00:12:24.729 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.write_buffer_size: 33554432 2024-09-17T00:12:24.730 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.max_write_buffer_number: 2 2024-09-17T00:12:24.730 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.compression: NoCompression 2024-09-17T00:12:24.730 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.bottommost_compression: Disabled 2024-09-17T00:12:24.730 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.prefix_extractor: nullptr 2024-09-17T00:12:24.730 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-17T00:12:24.730 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.num_levels: 7 2024-09-17T00:12:24.730 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-17T00:12:24.730 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-17T00:12:24.730 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-17T00:12:24.730 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-17T00:12:24.730 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-17T00:12:24.730 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-17T00:12:24.730 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-17T00:12:24.730 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-17T00:12:24.730 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-17T00:12:24.730 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-17T00:12:24.731 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-17T00:12:24.731 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.compression_opts.level: 32767 2024-09-17T00:12:24.731 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.compression_opts.strategy: 0 2024-09-17T00:12:24.731 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-17T00:12:24.731 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-17T00:12:24.731 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-17T00:12:24.731 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.compression_opts.enabled: false 2024-09-17T00:12:24.731 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-17T00:12:24.731 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-17T00:12:24.731 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-17T00:12:24.731 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.target_file_size_base: 67108864 2024-09-17T00:12:24.731 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-17T00:12:24.731 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-17T00:12:24.731 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-17T00:12:24.731 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-17T00:12:24.732 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-17T00:12:24.732 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-17T00:12:24.732 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-17T00:12:24.732 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-17T00:12:24.732 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-17T00:12:24.732 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-17T00:12:24.732 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-17T00:12:24.732 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-17T00:12:24.732 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-17T00:12:24.732 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.arena_block_size: 4194304 2024-09-17T00:12:24.732 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-17T00:12:24.732 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-17T00:12:24.732 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-09-17T00:12:24.732 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.disable_auto_compactions: 0 2024-09-17T00:12:24.733 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-17T00:12:24.733 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-17T00:12:24.733 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-17T00:12:24.733 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-17T00:12:24.733 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-17T00:12:24.733 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-17T00:12:24.733 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-17T00:12:24.733 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-17T00:12:24.733 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-17T00:12:24.733 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-17T00:12:24.733 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.table_properties_collectors: 2024-09-17T00:12:24.733 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.inplace_update_support: 0 2024-09-17T00:12:24.733 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-17T00:12:24.733 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-17T00:12:24.733 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-17T00:12:24.734 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-17T00:12:24.734 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.bloom_locality: 0 2024-09-17T00:12:24.734 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.max_successive_merges: 0 2024-09-17T00:12:24.734 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-17T00:12:24.734 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.paranoid_file_checks: 0 2024-09-17T00:12:24.734 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.force_consistency_checks: 1 2024-09-17T00:12:24.734 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.report_bg_io_stats: 0 2024-09-17T00:12:24.734 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.ttl: 2592000 2024-09-17T00:12:24.734 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-17T00:12:24.734 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.enable_blob_files: false 2024-09-17T00:12:24.734 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.min_blob_size: 0 2024-09-17T00:12:24.734 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.blob_file_size: 268435456 2024-09-17T00:12:24.734 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-17T00:12:24.734 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-17T00:12:24.735 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-17T00:12:24.735 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: [db/version_set.cc:4773] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000003 succeeded,manifest_file_number is 3, next_file_number is 5, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-09-17T00:12:24.735 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: [db/version_set.cc:4782] Column family [default] (ID 0), log number is 0 2024-09-17T00:12:24.735 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: [db/version_set.cc:4083] Creating manifest 7 2024-09-17T00:12:24.735 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726531944363812, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-09-17T00:12:24.735 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: [db/db_impl/db_impl_open.cc:847] Recovering log #4 mode 2 2024-09-17T00:12:24.735 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: [table/block_based/filter_policy.cc:996] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-09-17T00:12:24.735 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726531944364100, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1670, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1726531944, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "471f84f1-b3a3-4dc9-8118-4b47ca9953b6", "db_session_id": "OGVCPTSU3CQKCMZZA07P"}} 2024-09-17T00:12:24.735 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: [db/version_set.cc:4083] Creating manifest 9 2024-09-17T00:12:24.735 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726531944364402, "job": 1, "event": "recovery_finished"} 2024-09-17T00:12:24.735 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: [file/delete_scheduler.cc:73] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-17T00:12:24.735 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: [db/db_impl/db_impl_open.cc:1701] SstFileManager instance 0x559a3840ea80 2024-09-17T00:12:24.735 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: DB pointer 0x559a3841e000 2024-09-17T00:12:24.735 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: [db/db_impl/db_impl.cc:902] ------- DUMPING STATS ------- 2024-09-17T00:12:24.736 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-09-17T00:12:24.736 INFO:journalctl@ceph.mon.b.smithi099.stdout: ** DB Stats ** 2024-09-17T00:12:24.736 INFO:journalctl@ceph.mon.b.smithi099.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-17T00:12:24.736 INFO:journalctl@ceph.mon.b.smithi099.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-17T00:12:24.736 INFO:journalctl@ceph.mon.b.smithi099.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-17T00:12:24.736 INFO:journalctl@ceph.mon.b.smithi099.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-17T00:12:24.736 INFO:journalctl@ceph.mon.b.smithi099.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-17T00:12:24.736 INFO:journalctl@ceph.mon.b.smithi099.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-09-17T00:12:24.736 INFO:journalctl@ceph.mon.b.smithi099.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-17T00:12:24.736 INFO:journalctl@ceph.mon.b.smithi099.stdout: 2024-09-17T00:12:24.736 INFO:journalctl@ceph.mon.b.smithi099.stdout: ** Compaction Stats [default] ** 2024-09-17T00:12:24.736 INFO:journalctl@ceph.mon.b.smithi099.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-17T00:12:24.736 INFO:journalctl@ceph.mon.b.smithi099.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-17T00:12:24.736 INFO:journalctl@ceph.mon.b.smithi099.stdout: L0 1/0 1.63 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.7 0.00 0.00 1 0.000 0 0 2024-09-17T00:12:24.737 INFO:journalctl@ceph.mon.b.smithi099.stdout: Sum 1/0 1.63 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.7 0.00 0.00 1 0.000 0 0 2024-09-17T00:12:24.737 INFO:journalctl@ceph.mon.b.smithi099.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.7 0.00 0.00 1 0.000 0 0 2024-09-17T00:12:24.737 INFO:journalctl@ceph.mon.b.smithi099.stdout: 2024-09-17T00:12:24.737 INFO:journalctl@ceph.mon.b.smithi099.stdout: ** Compaction Stats [default] ** 2024-09-17T00:12:24.737 INFO:journalctl@ceph.mon.b.smithi099.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-17T00:12:24.737 INFO:journalctl@ceph.mon.b.smithi099.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-17T00:12:24.737 INFO:journalctl@ceph.mon.b.smithi099.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 5.7 0.00 0.00 1 0.000 0 0 2024-09-17T00:12:24.737 INFO:journalctl@ceph.mon.b.smithi099.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-17T00:12:24.737 INFO:journalctl@ceph.mon.b.smithi099.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-17T00:12:24.737 INFO:journalctl@ceph.mon.b.smithi099.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-17T00:12:24.737 INFO:journalctl@ceph.mon.b.smithi099.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-17T00:12:24.737 INFO:journalctl@ceph.mon.b.smithi099.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-17T00:12:24.737 INFO:journalctl@ceph.mon.b.smithi099.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-17T00:12:24.737 INFO:journalctl@ceph.mon.b.smithi099.stdout: Cumulative compaction: 0.00 GB write, 0.63 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-17T00:12:24.738 INFO:journalctl@ceph.mon.b.smithi099.stdout: Interval compaction: 0.00 GB write, 0.63 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-17T00:12:24.738 INFO:journalctl@ceph.mon.b.smithi099.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-17T00:12:24.738 INFO:journalctl@ceph.mon.b.smithi099.stdout: 2024-09-17T00:12:24.738 INFO:journalctl@ceph.mon.b.smithi099.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-17T00:12:24.738 INFO:journalctl@ceph.mon.b.smithi099.stdout: 2024-09-17T00:12:24.738 INFO:journalctl@ceph.mon.b.smithi099.stdout: ** Compaction Stats [default] ** 2024-09-17T00:12:24.738 INFO:journalctl@ceph.mon.b.smithi099.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-17T00:12:24.738 INFO:journalctl@ceph.mon.b.smithi099.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-17T00:12:24.738 INFO:journalctl@ceph.mon.b.smithi099.stdout: L0 1/0 1.63 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.7 0.00 0.00 1 0.000 0 0 2024-09-17T00:12:24.738 INFO:journalctl@ceph.mon.b.smithi099.stdout: Sum 1/0 1.63 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.7 0.00 0.00 1 0.000 0 0 2024-09-17T00:12:24.738 INFO:journalctl@ceph.mon.b.smithi099.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-09-17T00:12:24.738 INFO:journalctl@ceph.mon.b.smithi099.stdout: 2024-09-17T00:12:24.738 INFO:journalctl@ceph.mon.b.smithi099.stdout: ** Compaction Stats [default] ** 2024-09-17T00:12:24.738 INFO:journalctl@ceph.mon.b.smithi099.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-17T00:12:24.739 INFO:journalctl@ceph.mon.b.smithi099.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-17T00:12:24.739 INFO:journalctl@ceph.mon.b.smithi099.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 5.7 0.00 0.00 1 0.000 0 0 2024-09-17T00:12:24.739 INFO:journalctl@ceph.mon.b.smithi099.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-17T00:12:24.739 INFO:journalctl@ceph.mon.b.smithi099.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-17T00:12:24.739 INFO:journalctl@ceph.mon.b.smithi099.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-17T00:12:24.739 INFO:journalctl@ceph.mon.b.smithi099.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-17T00:12:24.739 INFO:journalctl@ceph.mon.b.smithi099.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-17T00:12:24.739 INFO:journalctl@ceph.mon.b.smithi099.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-17T00:12:24.739 INFO:journalctl@ceph.mon.b.smithi099.stdout: Cumulative compaction: 0.00 GB write, 0.62 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-17T00:12:24.739 INFO:journalctl@ceph.mon.b.smithi099.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-17T00:12:24.739 INFO:journalctl@ceph.mon.b.smithi099.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-17T00:12:24.739 INFO:journalctl@ceph.mon.b.smithi099.stdout: 2024-09-17T00:12:24.739 INFO:journalctl@ceph.mon.b.smithi099.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-17T00:12:24.739 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: mon.b does not exist in monmap, will attempt to join an existing cluster 2024-09-17T00:12:24.740 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: using public_addr v2:172.21.15.99:0/0 -> [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] 2024-09-17T00:12:24.740 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: starting mon.b rank -1 at public addrs [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] at bind addrs [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid 3aad27a2-7489-11ef-bceb-c7b262605968 2024-09-17T00:12:24.740 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: mon.b@-1(???) e0 preinit fsid 3aad27a2-7489-11ef-bceb-c7b262605968 2024-09-17T00:12:24.740 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: mon.b@-1(synchronizing).mds e1 new map 2024-09-17T00:12:24.740 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: mon.b@-1(synchronizing).mds e1 print_map 2024-09-17T00:12:24.740 INFO:journalctl@ceph.mon.b.smithi099.stdout: e1 2024-09-17T00:12:24.740 INFO:journalctl@ceph.mon.b.smithi099.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-17T00:12:24.740 INFO:journalctl@ceph.mon.b.smithi099.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-09-17T00:12:24.740 INFO:journalctl@ceph.mon.b.smithi099.stdout: legacy client fscid: -1 2024-09-17T00:12:24.740 INFO:journalctl@ceph.mon.b.smithi099.stdout: 2024-09-17T00:12:24.740 INFO:journalctl@ceph.mon.b.smithi099.stdout: No filesystems configured 2024-09-17T00:12:24.740 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-09-17T00:12:24.740 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-09-17T00:12:24.740 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-09-17T00:12:24.740 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-09-17T00:12:24.741 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-09-17T00:12:24.741 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-09-17T00:12:24.741 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-09-17T00:12:24.741 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-17T00:12:24.741 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-17T00:12:24.741 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-17T00:12:24.741 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi002:172.21.15.2=a;smithi002:[v2:172.21.15.2:3301,v1:172.21.15.2:6790]=c;smithi099:172.21.15.99=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:12:24.741 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: Saving service mon spec with placement smithi002:172.21.15.2=a;smithi002:[v2:172.21.15.2:3301,v1:172.21.15.2:6790]=c;smithi099:172.21.15.99=b;count:3 2024-09-17T00:12:24.741 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: Updating smithi099:/etc/ceph/ceph.conf 2024-09-17T00:12:24.741 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: Updating smithi099:/etc/ceph/ceph.client.admin.keyring 2024-09-17T00:12:24.741 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: Deploying daemon mon.c on smithi002 2024-09-17T00:12:24.741 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T00:12:24.741 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: from='client.? 172.21.15.99:0/252729467' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-17T00:12:24.741 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T00:12:24.741 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T00:12:24.742 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: mon.a calling monitor election 2024-09-17T00:12:24.742 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T00:12:24.742 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T00:12:24.742 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T00:12:24.742 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T00:12:24.742 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: mon.c calling monitor election 2024-09-17T00:12:24.742 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T00:12:24.742 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T00:12:24.742 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T00:12:24.742 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T00:12:24.742 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-09-17T00:12:24.742 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: monmap e2: 2 mons at {a=[v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0],c=[v2:172.21.15.2:3301/0,v1:172.21.15.2:6790/0]} 2024-09-17T00:12:24.742 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: fsmap 2024-09-17T00:12:24.742 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: osdmap e4: 0 total, 0 up, 0 in 2024-09-17T00:12:24.743 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: mgrmap e13: y(active, since 35s) 2024-09-17T00:12:24.743 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: overall HEALTH_OK 2024-09-17T00:12:24.743 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:12:24.743 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-17T00:12:24.743 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:12:24.743 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: Deploying daemon mon.b on smithi099 2024-09-17T00:12:24.743 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: from='client.? 172.21.15.99:0/4165018019' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-17T00:12:24.743 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T00:12:24.743 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T00:12:24.743 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T00:12:24.743 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-09-17T00:12:24.743 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:24 smithi099 ceph-mon[24594]: mon.b@-1(synchronizing) e2 handle_conf_change mon_allow_pool_delete,mon_cluster_log_to_file 2024-09-17T00:12:25.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:25 smithi099 bash[24539]: c6d2af7151872ee22f65b0ab7d833e966a67e7443f3369aa2a7ba9d93c7e9320 2024-09-17T00:12:25.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:25 smithi099 systemd[1]: Started Ceph mon.b for 3aad27a2-7489-11ef-bceb-c7b262605968. 2024-09-17T00:12:25.930 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-17T00:12:25.931 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3aad27a2-7489-11ef-bceb-c7b262605968 -- ceph mon dump -f json 2024-09-17T00:12:29.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:29 smithi002 ceph-mon[23737]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T00:12:29.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:29 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T00:12:29.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:29 smithi002 ceph-mon[23737]: mon.c calling monitor election 2024-09-17T00:12:29.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:29 smithi002 ceph-mon[23737]: mon.a calling monitor election 2024-09-17T00:12:29.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:29 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T00:12:29.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:29 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T00:12:29.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:29 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T00:12:29.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:29 smithi002 ceph-mon[23737]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T00:12:29.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:29 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T00:12:29.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:29 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T00:12:29.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:29 smithi002 ceph-mon[23737]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T00:12:29.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:29 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T00:12:29.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:29 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T00:12:29.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:29 smithi002 ceph-mon[23737]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-09-17T00:12:29.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:29 smithi002 ceph-mon[23737]: monmap e3: 3 mons at {a=[v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0],b=[v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0],c=[v2:172.21.15.2:3301/0,v1:172.21.15.2:6790/0]} 2024-09-17T00:12:29.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:29 smithi002 ceph-mon[23737]: fsmap 2024-09-17T00:12:29.731 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:29 smithi002 ceph-mon[23737]: osdmap e4: 0 total, 0 up, 0 in 2024-09-17T00:12:29.731 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:29 smithi002 ceph-mon[23737]: mgrmap e13: y(active, since 45s) 2024-09-17T00:12:29.731 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:29 smithi002 ceph-mon[23737]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-09-17T00:12:29.731 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:29 smithi002 ceph-mon[23737]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-09-17T00:12:29.731 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:29 smithi002 ceph-mon[23737]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-09-17T00:12:29.731 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:29 smithi002 ceph-mon[23737]: mon.b (rank 2) addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] is down (out of quorum) 2024-09-17T00:12:29.731 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:29 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:12:29.731 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:29 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T00:12:29.731 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:29 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:12:29.731 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:29 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:12:29.731 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:29 smithi002 ceph-mon[28203]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T00:12:29.731 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:29 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T00:12:29.732 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:29 smithi002 ceph-mon[28203]: mon.c calling monitor election 2024-09-17T00:12:29.732 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:29 smithi002 ceph-mon[28203]: mon.a calling monitor election 2024-09-17T00:12:29.732 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:29 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T00:12:29.732 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:29 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T00:12:29.732 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:29 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T00:12:29.732 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:29 smithi002 ceph-mon[28203]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T00:12:29.732 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:29 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T00:12:29.732 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:29 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T00:12:29.732 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:29 smithi002 ceph-mon[28203]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T00:12:29.732 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:29 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T00:12:29.732 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:29 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T00:12:29.732 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:29 smithi002 ceph-mon[28203]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-09-17T00:12:29.732 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:29 smithi002 ceph-mon[28203]: monmap e3: 3 mons at {a=[v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0],b=[v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0],c=[v2:172.21.15.2:3301/0,v1:172.21.15.2:6790/0]} 2024-09-17T00:12:29.732 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:29 smithi002 ceph-mon[28203]: fsmap 2024-09-17T00:12:29.733 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:29 smithi002 ceph-mon[28203]: osdmap e4: 0 total, 0 up, 0 in 2024-09-17T00:12:29.733 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:29 smithi002 ceph-mon[28203]: mgrmap e13: y(active, since 45s) 2024-09-17T00:12:29.733 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:29 smithi002 ceph-mon[28203]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-09-17T00:12:29.733 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:29 smithi002 ceph-mon[28203]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-09-17T00:12:29.733 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:29 smithi002 ceph-mon[28203]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-09-17T00:12:29.733 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:29 smithi002 ceph-mon[28203]: mon.b (rank 2) addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] is down (out of quorum) 2024-09-17T00:12:29.733 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:29 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:12:29.733 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:29 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T00:12:29.733 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:29 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:12:29.733 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:29 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:12:30.835 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-09-17T00:12:30.835 INFO:teuthology.orchestra.run.smithi099.stdout:{"epoch":3,"fsid":"3aad27a2-7489-11ef-bceb-c7b262605968","modified":"2024-09-17T00:12:24.385107Z","created":"2024-09-17T00:10:42.458448Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:3300","nonce":0},{"type":"v1","addr":"172.21.15.2:6789","nonce":0}]},"addr":"172.21.15.2:6789/0","public_addr":"172.21.15.2:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:3301","nonce":0},{"type":"v1","addr":"172.21.15.2:6790","nonce":0}]},"addr":"172.21.15.2:6790/0","public_addr":"172.21.15.2:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:3300","nonce":0},{"type":"v1","addr":"172.21.15.99:6789","nonce":0}]},"addr":"172.21.15.99:6789/0","public_addr":"172.21.15.99:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-09-17T00:12:30.836 INFO:teuthology.orchestra.run.smithi099.stderr:dumped monmap epoch 3 2024-09-17T00:12:31.406 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-09-17T00:12:31.406 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3aad27a2-7489-11ef-bceb-c7b262605968 -- ceph config generate-minimal-conf 2024-09-17T00:12:31.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:31 smithi002 ceph-mon[23737]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T00:12:31.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:31 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:12:31.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:31 smithi002 ceph-mon[23737]: Updating smithi099:/etc/ceph/ceph.conf 2024-09-17T00:12:31.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:31 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T00:12:31.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:31 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:12:31.480 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:31 smithi002 ceph-mon[23737]: Updating smithi002:/etc/ceph/ceph.conf 2024-09-17T00:12:31.480 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:31 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:12:31.480 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:31 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:12:31.480 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:31 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:12:31.480 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:31 smithi002 ceph-mon[23737]: Reconfiguring mon.a (unknown last config time)... 2024-09-17T00:12:31.480 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:31 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-17T00:12:31.480 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:31 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-17T00:12:31.480 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:31 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:12:31.480 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:31 smithi002 ceph-mon[23737]: Reconfiguring daemon mon.a on smithi002 2024-09-17T00:12:31.480 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:31 smithi002 ceph-mon[23737]: from='client.? 172.21.15.99:0/134293050' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-17T00:12:31.480 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:31 smithi002 ceph-mon[28203]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T00:12:31.481 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:31 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:12:31.481 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:31 smithi002 ceph-mon[28203]: Updating smithi099:/etc/ceph/ceph.conf 2024-09-17T00:12:31.481 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:31 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T00:12:31.481 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:31 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:12:31.481 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:31 smithi002 ceph-mon[28203]: Updating smithi002:/etc/ceph/ceph.conf 2024-09-17T00:12:31.481 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:31 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:12:31.481 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:31 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:12:31.481 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:31 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:12:31.481 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:31 smithi002 ceph-mon[28203]: Reconfiguring mon.a (unknown last config time)... 2024-09-17T00:12:31.481 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:31 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-17T00:12:31.481 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:31 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-17T00:12:31.481 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:31 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:12:31.482 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:31 smithi002 ceph-mon[28203]: Reconfiguring daemon mon.a on smithi002 2024-09-17T00:12:31.482 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:31 smithi002 ceph-mon[28203]: from='client.? 172.21.15.99:0/134293050' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-17T00:12:31.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:31 smithi099 ceph-mon[24594]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T00:12:31.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:31 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T00:12:31.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:31 smithi099 ceph-mon[24594]: mon.c calling monitor election 2024-09-17T00:12:31.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:31 smithi099 ceph-mon[24594]: mon.a calling monitor election 2024-09-17T00:12:31.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:31 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T00:12:31.718 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:31 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T00:12:31.718 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:31 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T00:12:31.718 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:31 smithi099 ceph-mon[24594]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T00:12:31.718 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:31 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T00:12:31.718 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:31 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T00:12:31.718 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:31 smithi099 ceph-mon[24594]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T00:12:31.718 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:31 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T00:12:31.718 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:31 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T00:12:31.718 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:31 smithi099 ceph-mon[24594]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-09-17T00:12:31.718 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:31 smithi099 ceph-mon[24594]: monmap e3: 3 mons at {a=[v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0],b=[v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0],c=[v2:172.21.15.2:3301/0,v1:172.21.15.2:6790/0]} 2024-09-17T00:12:31.718 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:31 smithi099 ceph-mon[24594]: fsmap 2024-09-17T00:12:31.718 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:31 smithi099 ceph-mon[24594]: osdmap e4: 0 total, 0 up, 0 in 2024-09-17T00:12:31.718 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:31 smithi099 ceph-mon[24594]: mgrmap e13: y(active, since 45s) 2024-09-17T00:12:31.718 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:31 smithi099 ceph-mon[24594]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-09-17T00:12:31.719 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:31 smithi099 ceph-mon[24594]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-09-17T00:12:31.719 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:31 smithi099 ceph-mon[24594]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-09-17T00:12:31.719 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:31 smithi099 ceph-mon[24594]: mon.b (rank 2) addr [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] is down (out of quorum) 2024-09-17T00:12:31.719 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:31 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:12:31.719 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:31 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T00:12:31.719 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:31 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:12:31.719 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:31 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:12:31.719 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:31 smithi099 ceph-mon[24594]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T00:12:31.719 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:31 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:12:31.719 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:31 smithi099 ceph-mon[24594]: Updating smithi099:/etc/ceph/ceph.conf 2024-09-17T00:12:31.719 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:31 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T00:12:31.719 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:31 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:12:31.719 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:31 smithi099 ceph-mon[24594]: Updating smithi002:/etc/ceph/ceph.conf 2024-09-17T00:12:31.720 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:31 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:12:31.720 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:31 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:12:31.720 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:31 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:12:31.720 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:31 smithi099 ceph-mon[24594]: Reconfiguring mon.a (unknown last config time)... 2024-09-17T00:12:31.720 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:31 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-17T00:12:31.720 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:31 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-17T00:12:31.720 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:31 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:12:31.720 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:31 smithi099 ceph-mon[24594]: Reconfiguring daemon mon.a on smithi002 2024-09-17T00:12:31.720 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:31 smithi099 ceph-mon[24594]: from='client.? 172.21.15.99:0/134293050' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-17T00:12:32.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:32 smithi002 ceph-mon[23737]: mon.b calling monitor election 2024-09-17T00:12:32.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:32 smithi002 ceph-mon[23737]: mon.b calling monitor election 2024-09-17T00:12:32.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:32 smithi002 ceph-mon[23737]: mon.c calling monitor election 2024-09-17T00:12:32.480 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:32 smithi002 ceph-mon[23737]: mon.a calling monitor election 2024-09-17T00:12:32.480 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:32 smithi002 ceph-mon[23737]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-09-17T00:12:32.480 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:32 smithi002 ceph-mon[23737]: monmap e3: 3 mons at {a=[v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0],b=[v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0],c=[v2:172.21.15.2:3301/0,v1:172.21.15.2:6790/0]} 2024-09-17T00:12:32.480 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:32 smithi002 ceph-mon[23737]: fsmap 2024-09-17T00:12:32.480 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:32 smithi002 ceph-mon[23737]: osdmap e4: 0 total, 0 up, 0 in 2024-09-17T00:12:32.480 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:32 smithi002 ceph-mon[23737]: mgrmap e13: y(active, since 47s) 2024-09-17T00:12:32.480 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:32 smithi002 ceph-mon[23737]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-09-17T00:12:32.480 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:32 smithi002 ceph-mon[23737]: Cluster is now healthy 2024-09-17T00:12:32.480 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:32 smithi002 ceph-mon[23737]: overall HEALTH_OK 2024-09-17T00:12:32.480 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:32 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:12:32.480 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:32 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-17T00:12:32.480 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:32 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-17T00:12:32.480 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:32 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:12:32.481 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:32 smithi002 ceph-mon[28203]: mon.b calling monitor election 2024-09-17T00:12:32.481 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:32 smithi002 ceph-mon[28203]: mon.b calling monitor election 2024-09-17T00:12:32.481 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:32 smithi002 ceph-mon[28203]: mon.c calling monitor election 2024-09-17T00:12:32.481 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:32 smithi002 ceph-mon[28203]: mon.a calling monitor election 2024-09-17T00:12:32.481 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:32 smithi002 ceph-mon[28203]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-09-17T00:12:32.481 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:32 smithi002 ceph-mon[28203]: monmap e3: 3 mons at {a=[v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0],b=[v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0],c=[v2:172.21.15.2:3301/0,v1:172.21.15.2:6790/0]} 2024-09-17T00:12:32.481 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:32 smithi002 ceph-mon[28203]: fsmap 2024-09-17T00:12:32.481 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:32 smithi002 ceph-mon[28203]: osdmap e4: 0 total, 0 up, 0 in 2024-09-17T00:12:32.481 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:32 smithi002 ceph-mon[28203]: mgrmap e13: y(active, since 47s) 2024-09-17T00:12:32.481 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:32 smithi002 ceph-mon[28203]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-09-17T00:12:32.481 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:32 smithi002 ceph-mon[28203]: Cluster is now healthy 2024-09-17T00:12:32.482 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:32 smithi002 ceph-mon[28203]: overall HEALTH_OK 2024-09-17T00:12:32.482 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:32 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:12:32.482 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:32 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-17T00:12:32.482 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:32 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-17T00:12:32.482 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:32 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:12:32.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:32 smithi099 ceph-mon[24594]: mon.b calling monitor election 2024-09-17T00:12:32.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:32 smithi099 ceph-mon[24594]: mon.b calling monitor election 2024-09-17T00:12:32.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:32 smithi099 ceph-mon[24594]: mon.c calling monitor election 2024-09-17T00:12:32.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:32 smithi099 ceph-mon[24594]: mon.a calling monitor election 2024-09-17T00:12:32.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:32 smithi099 ceph-mon[24594]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-09-17T00:12:32.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:32 smithi099 ceph-mon[24594]: monmap e3: 3 mons at {a=[v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0],b=[v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0],c=[v2:172.21.15.2:3301/0,v1:172.21.15.2:6790/0]} 2024-09-17T00:12:32.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:32 smithi099 ceph-mon[24594]: fsmap 2024-09-17T00:12:32.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:32 smithi099 ceph-mon[24594]: osdmap e4: 0 total, 0 up, 0 in 2024-09-17T00:12:32.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:32 smithi099 ceph-mon[24594]: mgrmap e13: y(active, since 47s) 2024-09-17T00:12:32.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:32 smithi099 ceph-mon[24594]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-09-17T00:12:32.718 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:32 smithi099 ceph-mon[24594]: Cluster is now healthy 2024-09-17T00:12:32.718 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:32 smithi099 ceph-mon[24594]: overall HEALTH_OK 2024-09-17T00:12:32.718 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:32 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:12:32.718 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:32 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-17T00:12:32.718 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:32 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-17T00:12:32.718 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:32 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:12:33.557 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:33 smithi002 ceph-mon[23737]: Reconfiguring mon.c (monmap changed)... 2024-09-17T00:12:33.557 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:33 smithi002 ceph-mon[23737]: Reconfiguring daemon mon.c on smithi002 2024-09-17T00:12:33.557 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:33 smithi002 ceph-mon[23737]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T00:12:33.557 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:33 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T00:12:33.557 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:33 smithi002 ceph-mon[28203]: Reconfiguring mon.c (monmap changed)... 2024-09-17T00:12:33.557 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:33 smithi002 ceph-mon[28203]: Reconfiguring daemon mon.c on smithi002 2024-09-17T00:12:33.558 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:33 smithi002 ceph-mon[28203]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T00:12:33.558 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:33 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T00:12:33.716 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:33 smithi099 ceph-mon[24594]: Reconfiguring mon.c (monmap changed)... 2024-09-17T00:12:33.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:33 smithi099 ceph-mon[24594]: Reconfiguring daemon mon.c on smithi002 2024-09-17T00:12:33.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:33 smithi099 ceph-mon[24594]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T00:12:33.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:33 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T00:12:33.846 INFO:teuthology.orchestra.run.smithi002.stdout:# minimal ceph.conf for 3aad27a2-7489-11ef-bceb-c7b262605968 2024-09-17T00:12:33.846 INFO:teuthology.orchestra.run.smithi002.stdout:[global] 2024-09-17T00:12:33.846 INFO:teuthology.orchestra.run.smithi002.stdout: fsid = 3aad27a2-7489-11ef-bceb-c7b262605968 2024-09-17T00:12:33.846 INFO:teuthology.orchestra.run.smithi002.stdout: mon_host = [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] [v2:172.21.15.2:3301/0,v1:172.21.15.2:6790/0] 2024-09-17T00:12:34.637 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:34 smithi002 ceph-mon[23737]: from='client.? 172.21.15.2:0/1786141381' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:12:34.638 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:34 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:12:34.638 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:34 smithi002 ceph-mon[28203]: from='client.? 172.21.15.2:0/1786141381' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:12:34.638 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:34 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:12:34.667 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-09-17T00:12:34.668 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-09-17T00:12:34.668 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd of=/etc/ceph/ceph.conf 2024-09-17T00:12:34.699 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-09-17T00:12:34.699 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-17T00:12:34.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:34 smithi099 ceph-mon[24594]: from='client.? 172.21.15.2:0/1786141381' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:12:34.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:34 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:12:34.765 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-09-17T00:12:34.766 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd of=/etc/ceph/ceph.conf 2024-09-17T00:12:34.797 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-09-17T00:12:34.798 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-17T00:12:34.866 INFO:tasks.cephadm:Adding mgr.y on smithi002 2024-09-17T00:12:34.866 INFO:tasks.cephadm:Adding mgr.x on smithi099 2024-09-17T00:12:34.867 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3aad27a2-7489-11ef-bceb-c7b262605968 -- ceph orch apply mgr '2;smithi002=y;smithi099=x' 2024-09-17T00:12:35.392 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:35 smithi099 ceph-mon[24594]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T00:12:35.393 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:35 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:12:35.393 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:35 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-17T00:12:35.393 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:35 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-17T00:12:35.393 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:35 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:12:35.732 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:35 smithi002 ceph-mon[23737]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T00:12:35.732 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:35 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:12:35.733 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:35 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-17T00:12:35.733 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:35 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-17T00:12:35.733 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:35 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:12:35.733 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:35 smithi002 ceph-mon[28203]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T00:12:35.733 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:35 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:12:35.733 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:35 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-17T00:12:35.733 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:35 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-17T00:12:35.733 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:35 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:12:36.328 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:36 smithi099 ceph-mon[24594]: Reconfiguring mon.b (monmap changed)... 2024-09-17T00:12:36.328 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:36 smithi099 ceph-mon[24594]: Reconfiguring daemon mon.b on smithi099 2024-09-17T00:12:36.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:36 smithi002 ceph-mon[23737]: Reconfiguring mon.b (monmap changed)... 2024-09-17T00:12:36.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:36 smithi002 ceph-mon[23737]: Reconfiguring daemon mon.b on smithi099 2024-09-17T00:12:36.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:36 smithi002 ceph-mon[28203]: Reconfiguring mon.b (monmap changed)... 2024-09-17T00:12:36.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:36 smithi002 ceph-mon[28203]: Reconfiguring daemon mon.b on smithi099 2024-09-17T00:12:36.858 INFO:teuthology.orchestra.run.smithi099.stdout:Scheduled mgr update... 2024-09-17T00:12:37.380 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:37 smithi099 ceph-mon[24594]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T00:12:37.381 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:37 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:12:37.620 DEBUG:teuthology.orchestra.run.smithi099:mgr.x> sudo journalctl -f -n 0 -u ceph-3aad27a2-7489-11ef-bceb-c7b262605968@mgr.x.service 2024-09-17T00:12:37.623 INFO:tasks.cephadm:Deploying OSDs... 2024-09-17T00:12:37.623 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-09-17T00:12:37.623 DEBUG:teuthology.orchestra.run.smithi002:> dd if=/scratch_devs of=/dev/stdout 2024-09-17T00:12:37.643 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-17T00:12:37.644 DEBUG:teuthology.orchestra.run.smithi002:> stat /dev/vg_nvme/lv_1 2024-09-17T00:12:37.704 INFO:teuthology.orchestra.run.smithi002.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-17T00:12:37.704 INFO:teuthology.orchestra.run.smithi002.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-17T00:12:37.704 INFO:teuthology.orchestra.run.smithi002.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-09-17T00:12:37.704 INFO:teuthology.orchestra.run.smithi002.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-17T00:12:37.704 INFO:teuthology.orchestra.run.smithi002.stdout:Context: system_u:object_r:device_t:s0 2024-09-17T00:12:37.704 INFO:teuthology.orchestra.run.smithi002.stdout:Access: 2024-09-17 00:12:14.112066953 +0000 2024-09-17T00:12:37.704 INFO:teuthology.orchestra.run.smithi002.stdout:Modify: 2024-09-17 00:10:01.662933686 +0000 2024-09-17T00:12:37.704 INFO:teuthology.orchestra.run.smithi002.stdout:Change: 2024-09-17 00:10:01.662933686 +0000 2024-09-17T00:12:37.704 INFO:teuthology.orchestra.run.smithi002.stdout: Birth: 2024-09-17 00:10:01.662933686 +0000 2024-09-17T00:12:37.705 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-17T00:12:37.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:37 smithi002 ceph-mon[23737]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T00:12:37.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:37 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:12:37.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:37 smithi002 ceph-mon[28203]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T00:12:37.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:37 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:12:37.758 INFO:teuthology.orchestra.run.smithi002.stderr:1+0 records in 2024-09-17T00:12:37.759 INFO:teuthology.orchestra.run.smithi002.stderr:1+0 records out 2024-09-17T00:12:37.759 INFO:teuthology.orchestra.run.smithi002.stderr:512 bytes copied, 0.00013049 s, 3.9 MB/s 2024-09-17T00:12:37.760 DEBUG:teuthology.orchestra.run.smithi002:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-17T00:12:37.820 DEBUG:teuthology.orchestra.run.smithi002:> stat /dev/vg_nvme/lv_2 2024-09-17T00:12:37.878 INFO:teuthology.orchestra.run.smithi002.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-17T00:12:37.878 INFO:teuthology.orchestra.run.smithi002.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-17T00:12:37.878 INFO:teuthology.orchestra.run.smithi002.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-09-17T00:12:37.878 INFO:teuthology.orchestra.run.smithi002.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-17T00:12:37.878 INFO:teuthology.orchestra.run.smithi002.stdout:Context: system_u:object_r:device_t:s0 2024-09-17T00:12:37.878 INFO:teuthology.orchestra.run.smithi002.stdout:Access: 2024-09-17 00:12:14.113066932 +0000 2024-09-17T00:12:37.878 INFO:teuthology.orchestra.run.smithi002.stdout:Modify: 2024-09-17 00:10:01.984926951 +0000 2024-09-17T00:12:37.879 INFO:teuthology.orchestra.run.smithi002.stdout:Change: 2024-09-17 00:10:01.984926951 +0000 2024-09-17T00:12:37.879 INFO:teuthology.orchestra.run.smithi002.stdout: Birth: 2024-09-17 00:10:01.984926951 +0000 2024-09-17T00:12:37.879 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-17T00:12:37.945 INFO:teuthology.orchestra.run.smithi002.stderr:1+0 records in 2024-09-17T00:12:37.945 INFO:teuthology.orchestra.run.smithi002.stderr:1+0 records out 2024-09-17T00:12:37.945 INFO:teuthology.orchestra.run.smithi002.stderr:512 bytes copied, 0.000118347 s, 4.3 MB/s 2024-09-17T00:12:37.946 DEBUG:teuthology.orchestra.run.smithi002:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-17T00:12:38.005 DEBUG:teuthology.orchestra.run.smithi002:> stat /dev/vg_nvme/lv_3 2024-09-17T00:12:38.065 INFO:teuthology.orchestra.run.smithi002.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-17T00:12:38.065 INFO:teuthology.orchestra.run.smithi002.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-17T00:12:38.065 INFO:teuthology.orchestra.run.smithi002.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-09-17T00:12:38.065 INFO:teuthology.orchestra.run.smithi002.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-17T00:12:38.065 INFO:teuthology.orchestra.run.smithi002.stdout:Context: system_u:object_r:device_t:s0 2024-09-17T00:12:38.065 INFO:teuthology.orchestra.run.smithi002.stdout:Access: 2024-09-17 00:12:14.113066932 +0000 2024-09-17T00:12:38.065 INFO:teuthology.orchestra.run.smithi002.stdout:Modify: 2024-09-17 00:10:02.322919882 +0000 2024-09-17T00:12:38.065 INFO:teuthology.orchestra.run.smithi002.stdout:Change: 2024-09-17 00:10:02.322919882 +0000 2024-09-17T00:12:38.065 INFO:teuthology.orchestra.run.smithi002.stdout: Birth: 2024-09-17 00:10:02.322919882 +0000 2024-09-17T00:12:38.066 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-17T00:12:38.131 INFO:teuthology.orchestra.run.smithi002.stderr:1+0 records in 2024-09-17T00:12:38.131 INFO:teuthology.orchestra.run.smithi002.stderr:1+0 records out 2024-09-17T00:12:38.132 INFO:teuthology.orchestra.run.smithi002.stderr:512 bytes copied, 0.000111364 s, 4.6 MB/s 2024-09-17T00:12:38.133 DEBUG:teuthology.orchestra.run.smithi002:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-17T00:12:38.190 DEBUG:teuthology.orchestra.run.smithi002:> stat /dev/vg_nvme/lv_4 2024-09-17T00:12:38.247 INFO:teuthology.orchestra.run.smithi002.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-17T00:12:38.247 INFO:teuthology.orchestra.run.smithi002.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-17T00:12:38.248 INFO:teuthology.orchestra.run.smithi002.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-09-17T00:12:38.248 INFO:teuthology.orchestra.run.smithi002.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-17T00:12:38.248 INFO:teuthology.orchestra.run.smithi002.stdout:Context: system_u:object_r:device_t:s0 2024-09-17T00:12:38.248 INFO:teuthology.orchestra.run.smithi002.stdout:Access: 2024-09-17 00:12:14.113066932 +0000 2024-09-17T00:12:38.248 INFO:teuthology.orchestra.run.smithi002.stdout:Modify: 2024-09-17 00:10:02.647913085 +0000 2024-09-17T00:12:38.248 INFO:teuthology.orchestra.run.smithi002.stdout:Change: 2024-09-17 00:10:02.647913085 +0000 2024-09-17T00:12:38.248 INFO:teuthology.orchestra.run.smithi002.stdout: Birth: 2024-09-17 00:10:02.647913085 +0000 2024-09-17T00:12:38.248 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-17T00:12:38.313 INFO:teuthology.orchestra.run.smithi002.stderr:1+0 records in 2024-09-17T00:12:38.313 INFO:teuthology.orchestra.run.smithi002.stderr:1+0 records out 2024-09-17T00:12:38.313 INFO:teuthology.orchestra.run.smithi002.stderr:512 bytes copied, 9.5941e-05 s, 5.3 MB/s 2024-09-17T00:12:38.314 DEBUG:teuthology.orchestra.run.smithi002:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-17T00:12:38.372 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-09-17T00:12:38.372 DEBUG:teuthology.orchestra.run.smithi099:> dd if=/scratch_devs of=/dev/stdout 2024-09-17T00:12:38.389 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-17T00:12:38.389 DEBUG:teuthology.orchestra.run.smithi099:> stat /dev/vg_nvme/lv_1 2024-09-17T00:12:38.445 INFO:teuthology.orchestra.run.smithi099.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-17T00:12:38.445 INFO:teuthology.orchestra.run.smithi099.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-17T00:12:38.445 INFO:teuthology.orchestra.run.smithi099.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-09-17T00:12:38.445 INFO:teuthology.orchestra.run.smithi099.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-17T00:12:38.445 INFO:teuthology.orchestra.run.smithi099.stdout:Context: system_u:object_r:device_t:s0 2024-09-17T00:12:38.445 INFO:teuthology.orchestra.run.smithi099.stdout:Access: 2024-09-17 00:12:23.278501572 +0000 2024-09-17T00:12:38.445 INFO:teuthology.orchestra.run.smithi099.stdout:Modify: 2024-09-17 00:09:55.231930256 +0000 2024-09-17T00:12:38.446 INFO:teuthology.orchestra.run.smithi099.stdout:Change: 2024-09-17 00:09:55.231930256 +0000 2024-09-17T00:12:38.446 INFO:teuthology.orchestra.run.smithi099.stdout: Birth: 2024-09-17 00:09:55.231930256 +0000 2024-09-17T00:12:38.446 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-17T00:12:38.508 INFO:teuthology.orchestra.run.smithi099.stderr:1+0 records in 2024-09-17T00:12:38.508 INFO:teuthology.orchestra.run.smithi099.stderr:1+0 records out 2024-09-17T00:12:38.508 INFO:teuthology.orchestra.run.smithi099.stderr:512 bytes copied, 0.000210223 s, 2.4 MB/s 2024-09-17T00:12:38.509 DEBUG:teuthology.orchestra.run.smithi099:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-17T00:12:38.566 DEBUG:teuthology.orchestra.run.smithi099:> stat /dev/vg_nvme/lv_2 2024-09-17T00:12:38.621 INFO:teuthology.orchestra.run.smithi099.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-17T00:12:38.622 INFO:teuthology.orchestra.run.smithi099.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-17T00:12:38.622 INFO:teuthology.orchestra.run.smithi099.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-09-17T00:12:38.622 INFO:teuthology.orchestra.run.smithi099.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-17T00:12:38.622 INFO:teuthology.orchestra.run.smithi099.stdout:Context: system_u:object_r:device_t:s0 2024-09-17T00:12:38.622 INFO:teuthology.orchestra.run.smithi099.stdout:Access: 2024-09-17 00:12:23.278501572 +0000 2024-09-17T00:12:38.622 INFO:teuthology.orchestra.run.smithi099.stdout:Modify: 2024-09-17 00:09:55.557923046 +0000 2024-09-17T00:12:38.622 INFO:teuthology.orchestra.run.smithi099.stdout:Change: 2024-09-17 00:09:55.557923046 +0000 2024-09-17T00:12:38.622 INFO:teuthology.orchestra.run.smithi099.stdout: Birth: 2024-09-17 00:09:55.557923046 +0000 2024-09-17T00:12:38.622 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-17T00:12:38.684 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:38 smithi099 ceph-mon[24594]: from='client.24110 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi002=y;smithi099=x", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:12:38.684 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:38 smithi099 ceph-mon[24594]: Saving service mgr spec with placement smithi002=y;smithi099=x;count:2 2024-09-17T00:12:38.684 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:38 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:12:38.684 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:38 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T00:12:38.684 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:38 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:12:38.685 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:38 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:12:38.685 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:38 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:12:38.685 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:38 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-17T00:12:38.685 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:38 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-17T00:12:38.685 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:38 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-17T00:12:38.685 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:38 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:12:38.686 INFO:teuthology.orchestra.run.smithi099.stderr:1+0 records in 2024-09-17T00:12:38.686 INFO:teuthology.orchestra.run.smithi099.stderr:1+0 records out 2024-09-17T00:12:38.686 INFO:teuthology.orchestra.run.smithi099.stderr:512 bytes copied, 0.000200645 s, 2.6 MB/s 2024-09-17T00:12:38.688 DEBUG:teuthology.orchestra.run.smithi099:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-17T00:12:38.744 DEBUG:teuthology.orchestra.run.smithi099:> stat /dev/vg_nvme/lv_3 2024-09-17T00:12:38.800 INFO:teuthology.orchestra.run.smithi099.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-17T00:12:38.800 INFO:teuthology.orchestra.run.smithi099.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-17T00:12:38.801 INFO:teuthology.orchestra.run.smithi099.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-09-17T00:12:38.801 INFO:teuthology.orchestra.run.smithi099.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-17T00:12:38.801 INFO:teuthology.orchestra.run.smithi099.stdout:Context: system_u:object_r:device_t:s0 2024-09-17T00:12:38.801 INFO:teuthology.orchestra.run.smithi099.stdout:Access: 2024-09-17 00:12:23.279501550 +0000 2024-09-17T00:12:38.801 INFO:teuthology.orchestra.run.smithi099.stdout:Modify: 2024-09-17 00:09:55.879915926 +0000 2024-09-17T00:12:38.801 INFO:teuthology.orchestra.run.smithi099.stdout:Change: 2024-09-17 00:09:55.879915926 +0000 2024-09-17T00:12:38.801 INFO:teuthology.orchestra.run.smithi099.stdout: Birth: 2024-09-17 00:09:55.879915926 +0000 2024-09-17T00:12:38.801 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-17T00:12:38.864 INFO:teuthology.orchestra.run.smithi099.stderr:1+0 records in 2024-09-17T00:12:38.864 INFO:teuthology.orchestra.run.smithi099.stderr:1+0 records out 2024-09-17T00:12:38.864 INFO:teuthology.orchestra.run.smithi099.stderr:512 bytes copied, 0.000208446 s, 2.5 MB/s 2024-09-17T00:12:38.865 DEBUG:teuthology.orchestra.run.smithi099:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-17T00:12:38.922 DEBUG:teuthology.orchestra.run.smithi099:> stat /dev/vg_nvme/lv_4 2024-09-17T00:12:38.977 INFO:teuthology.orchestra.run.smithi099.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-17T00:12:38.977 INFO:teuthology.orchestra.run.smithi099.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-17T00:12:38.977 INFO:teuthology.orchestra.run.smithi099.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-09-17T00:12:38.977 INFO:teuthology.orchestra.run.smithi099.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-17T00:12:38.977 INFO:teuthology.orchestra.run.smithi099.stdout:Context: system_u:object_r:device_t:s0 2024-09-17T00:12:38.977 INFO:teuthology.orchestra.run.smithi099.stdout:Access: 2024-09-17 00:12:23.279501550 +0000 2024-09-17T00:12:38.977 INFO:teuthology.orchestra.run.smithi099.stdout:Modify: 2024-09-17 00:09:56.214908518 +0000 2024-09-17T00:12:38.977 INFO:teuthology.orchestra.run.smithi099.stdout:Change: 2024-09-17 00:09:56.214908518 +0000 2024-09-17T00:12:38.977 INFO:teuthology.orchestra.run.smithi099.stdout: Birth: 2024-09-17 00:09:56.214908518 +0000 2024-09-17T00:12:38.978 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-17T00:12:38.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:38 smithi002 ceph-mon[23737]: from='client.24110 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi002=y;smithi099=x", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:12:38.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:38 smithi002 ceph-mon[23737]: Saving service mgr spec with placement smithi002=y;smithi099=x;count:2 2024-09-17T00:12:38.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:38 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:12:38.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:38 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T00:12:38.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:38 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:12:38.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:38 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:12:38.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:38 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:12:38.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:38 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-17T00:12:38.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:38 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-17T00:12:38.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:38 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-17T00:12:38.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:38 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:12:38.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:38 smithi002 ceph-mon[28203]: from='client.24110 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi002=y;smithi099=x", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:12:38.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:38 smithi002 ceph-mon[28203]: Saving service mgr spec with placement smithi002=y;smithi099=x;count:2 2024-09-17T00:12:38.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:38 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:12:38.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:38 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T00:12:38.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:38 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:12:38.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:38 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:12:38.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:38 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:12:38.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:38 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-17T00:12:38.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:38 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-17T00:12:38.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:38 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-17T00:12:38.981 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:38 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:12:39.040 INFO:teuthology.orchestra.run.smithi099.stderr:1+0 records in 2024-09-17T00:12:39.041 INFO:teuthology.orchestra.run.smithi099.stderr:1+0 records out 2024-09-17T00:12:39.041 INFO:teuthology.orchestra.run.smithi099.stderr:512 bytes copied, 0.000200345 s, 2.6 MB/s 2024-09-17T00:12:39.042 DEBUG:teuthology.orchestra.run.smithi099:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-17T00:12:39.098 INFO:tasks.cephadm:Deploying osd.0 on smithi002 with /dev/vg_nvme/lv_4... 2024-09-17T00:12:39.098 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3aad27a2-7489-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_4 2024-09-17T00:12:39.689 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:39 smithi099 ceph-mon[24594]: Deploying daemon mgr.x on smithi099 2024-09-17T00:12:39.689 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:39 smithi099 ceph-mon[24594]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T00:12:39.718 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:39 smithi002 ceph-mon[28203]: Deploying daemon mgr.x on smithi099 2024-09-17T00:12:39.718 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:39 smithi002 ceph-mon[28203]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T00:12:39.718 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:39 smithi002 ceph-mon[23737]: Deploying daemon mgr.x on smithi099 2024-09-17T00:12:39.718 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:39 smithi002 ceph-mon[23737]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T00:12:39.967 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:12:39 smithi099 podman[25757]: 2024-09-17 00:12:39.918609931 +0000 UTC m=+0.129122311 container create 9db4cd4717a560ad4ae1fd9ae5446c6a9198d440b7ac1a04807a7d4946f8b47e (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x, io.openshift.tags=base centos centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., RELEASE=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, build-date=2022-05-03T08:36:31.336870, release=754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, CEPH_POINT_RELEASE=-17.2.0, maintainer=Guillaume Abrioux , architecture=x86_64, io.k8s.display-name=CentOS Stream 8, vcs-type=git, GIT_CLEAN=True, io.buildah.version=1.19.8, com.redhat.component=centos-stream-container, ceph=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vendor=Red Hat, Inc., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, distribution-scope=public, name=centos-stream, version=8, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_BRANCH=HEAD) 2024-09-17T00:12:40.261 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:12:40 smithi099 podman[25757]: 2024-09-17 00:12:40.137855027 +0000 UTC m=+0.348367422 container init 9db4cd4717a560ad4ae1fd9ae5446c6a9198d440b7ac1a04807a7d4946f8b47e (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x, io.buildah.version=1.19.8, io.k8s.display-name=CentOS Stream 8, maintainer=Guillaume Abrioux , io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, name=centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.tags=base centos centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.component=centos-stream-container, ceph=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_CLEAN=True, build-date=2022-05-03T08:36:31.336870, CEPH_POINT_RELEASE=-17.2.0, version=8, vcs-type=git, GIT_BRANCH=HEAD, release=754, distribution-scope=public, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, architecture=x86_64, io.openshift.expose-services=, RELEASE=HEAD) 2024-09-17T00:12:40.261 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:12:40 smithi099 podman[25757]: 2024-09-17 00:12:40.140573877 +0000 UTC m=+0.351086256 container start 9db4cd4717a560ad4ae1fd9ae5446c6a9198d440b7ac1a04807a7d4946f8b47e (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, distribution-scope=public, version=8, name=centos-stream, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, maintainer=Guillaume Abrioux , io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.expose-services=, vcs-type=git, CEPH_POINT_RELEASE=-17.2.0, GIT_CLEAN=True, RELEASE=HEAD, build-date=2022-05-03T08:36:31.336870, vendor=Red Hat, Inc., io.buildah.version=1.19.8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, release=754, ceph=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.tags=base centos centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.display-name=CentOS Stream 8, GIT_BRANCH=HEAD, com.redhat.component=centos-stream-container) 2024-09-17T00:12:40.261 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:12:40 smithi099 ceph-mgr[25776]: -- 172.21.15.99:0/166774664 <== mon.2 v2:172.21.15.99:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55cf57d6a340 con 0x55cf57d43000 2024-09-17T00:12:40.261 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:12:40 smithi099 bash[25757]: 9db4cd4717a560ad4ae1fd9ae5446c6a9198d440b7ac1a04807a7d4946f8b47e 2024-09-17T00:12:40.572 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:12:40 smithi099 systemd[1]: Started Ceph mgr.x for 3aad27a2-7489-11ef-bceb-c7b262605968. 2024-09-17T00:12:40.573 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:12:40 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[25772]: 2024-09-17T00:12:40.280+0000 7f38cd75f000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-17T00:12:40.573 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:12:40 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[25772]: 2024-09-17T00:12:40.369+0000 7f38cd75f000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-17T00:12:40.573 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:12:40 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[25772]: 2024-09-17T00:12:40.574+0000 7f38cd75f000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-17T00:12:40.914 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:12:40 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[25772]: 2024-09-17T00:12:40.671+0000 7f38cd75f000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-17T00:12:40.914 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:12:40 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[25772]: 2024-09-17T00:12:40.804+0000 7f38cd75f000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-17T00:12:41.217 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:12:41 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[25772]: 2024-09-17T00:12:41.017+0000 7f38cd75f000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-17T00:12:41.430 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:41 smithi002 ceph-mon[23737]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T00:12:41.431 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:41 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:12:41.431 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:41 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T00:12:41.431 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:41 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:12:41.431 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:41 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:12:41.431 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:41 smithi002 ceph-mon[28203]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T00:12:41.431 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:41 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:12:41.431 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:41 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T00:12:41.431 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:41 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:12:41.431 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:41 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:12:41.613 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:12:41 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[25772]: 2024-09-17T00:12:41.363+0000 7f38cd75f000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-17T00:12:41.613 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:12:41 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[25772]: 2024-09-17T00:12:41.452+0000 7f38cd75f000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-17T00:12:41.613 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:12:41 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[25772]: 2024-09-17T00:12:41.615+0000 7f38cd75f000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-17T00:12:41.613 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:41 smithi099 ceph-mon[24594]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T00:12:41.613 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:41 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:12:41.613 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:41 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T00:12:41.614 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:41 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:12:41.614 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:41 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:12:41.766 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-17T00:12:41.787 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3aad27a2-7489-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi002:vg_nvme/lv_4 2024-09-17T00:12:42.650 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:42 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:12:42.650 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:42 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:12:42.650 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:42 smithi099 ceph-mon[24594]: Reconfiguring mgr.y (unknown last config time)... 2024-09-17T00:12:42.651 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:42 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-17T00:12:42.651 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:42 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-17T00:12:42.651 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:42 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:12:42.651 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:42 smithi099 ceph-mon[24594]: Reconfiguring daemon mgr.y on smithi002 2024-09-17T00:12:42.675 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:42 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:12:42.676 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:42 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:12:42.676 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:42 smithi002 ceph-mon[28203]: Reconfiguring mgr.y (unknown last config time)... 2024-09-17T00:12:42.676 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:42 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-17T00:12:42.676 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:42 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-17T00:12:42.676 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:42 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:12:42.676 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:42 smithi002 ceph-mon[28203]: Reconfiguring daemon mgr.y on smithi002 2024-09-17T00:12:42.676 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:42 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:12:42.676 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:42 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:12:42.676 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:42 smithi002 ceph-mon[23737]: Reconfiguring mgr.y (unknown last config time)... 2024-09-17T00:12:42.676 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:42 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-17T00:12:42.676 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:42 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-17T00:12:42.677 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:42 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:12:42.677 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:42 smithi002 ceph-mon[23737]: Reconfiguring daemon mgr.y on smithi002 2024-09-17T00:12:42.966 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:12:42 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[25772]: 2024-09-17T00:12:42.653+0000 7f38cd75f000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-17T00:12:42.967 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:12:42 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[25772]: 2024-09-17T00:12:42.742+0000 7f38cd75f000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-17T00:12:43.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:43 smithi002 ceph-mon[28203]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T00:12:43.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:43 smithi002 ceph-mon[23737]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T00:12:43.928 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:43 smithi099 ceph-mon[24594]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T00:12:43.928 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:12:43 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[25772]: 2024-09-17T00:12:43.627+0000 7f38cd75f000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-17T00:12:44.217 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:12:43 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[25772]: 2024-09-17T00:12:43.930+0000 7f38cd75f000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-17T00:12:44.217 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:12:44 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[25772]: 2024-09-17T00:12:44.061+0000 7f38cd75f000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-17T00:12:44.217 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:12:44 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[25772]: 2024-09-17T00:12:44.147+0000 7f38cd75f000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-17T00:12:44.537 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:12:44 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[25772]: 2024-09-17T00:12:44.245+0000 7f38cd75f000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-17T00:12:44.537 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:12:44 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[25772]: 2024-09-17T00:12:44.331+0000 7f38cd75f000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-17T00:12:44.537 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:12:44 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[25772]: 2024-09-17T00:12:44.477+0000 7f38cd75f000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-17T00:12:44.942 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:12:44 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[25772]: 2024-09-17T00:12:44.571+0000 7f38cd75f000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-17T00:12:44.942 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:12:44 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[25772]: 2024-09-17T00:12:44.665+0000 7f38cd75f000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-17T00:12:44.942 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:44 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:12:44.942 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:44 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T00:12:44.942 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:44 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:12:44.942 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:44 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:12:44.942 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:44 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:12:44.943 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:44 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T00:12:44.943 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:44 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:12:44.943 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:44 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T00:12:44.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:44 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:12:44.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:44 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T00:12:44.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:44 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:12:44.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:44 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:12:44.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:44 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:12:44.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:44 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T00:12:44.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:44 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:12:44.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:44 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T00:12:44.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:44 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:12:44.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:44 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T00:12:44.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:44 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:12:44.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:44 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:12:44.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:44 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:12:44.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:44 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T00:12:44.981 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:44 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:12:44.981 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:44 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T00:12:45.217 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:12:44 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[25772]: 2024-09-17T00:12:44.944+0000 7f38cd75f000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-17T00:12:45.217 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:12:45 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[25772]: 2024-09-17T00:12:45.031+0000 7f38cd75f000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-17T00:12:45.217 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:12:45 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[25772]: 2024-09-17T00:12:45.119+0000 7f38cd75f000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-17T00:12:45.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:45 smithi002 ceph-mon[28203]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T00:12:45.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:45 smithi002 ceph-mon[28203]: from='client.24124 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi002:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:12:45.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:45 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-17T00:12:45.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:45 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-17T00:12:45.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:45 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:12:45.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:45 smithi002 ceph-mon[23737]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T00:12:45.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:45 smithi002 ceph-mon[23737]: from='client.24124 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi002:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:12:45.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:45 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-17T00:12:45.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:45 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-17T00:12:45.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:45 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:12:45.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:45 smithi099 ceph-mon[24594]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T00:12:45.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:45 smithi099 ceph-mon[24594]: from='client.24124 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi002:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:12:45.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:45 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-17T00:12:45.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:45 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-17T00:12:45.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:45 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:12:46.467 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:12:46 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[25772]: 2024-09-17T00:12:46.167+0000 7f38cd75f000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-17T00:12:46.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:46 smithi002 ceph-mon[23737]: Standby manager daemon x started 2024-09-17T00:12:46.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:46 smithi002 ceph-mon[23737]: from='mgr.? 172.21.15.99:0/2511772451' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-17T00:12:46.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:46 smithi002 ceph-mon[23737]: from='mgr.? 172.21.15.99:0/2511772451' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-17T00:12:46.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:46 smithi002 ceph-mon[23737]: from='mgr.? 172.21.15.99:0/2511772451' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-17T00:12:46.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:46 smithi002 ceph-mon[23737]: from='mgr.? 172.21.15.99:0/2511772451' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-17T00:12:46.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:46 smithi002 ceph-mon[28203]: Standby manager daemon x started 2024-09-17T00:12:46.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:46 smithi002 ceph-mon[28203]: from='mgr.? 172.21.15.99:0/2511772451' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-17T00:12:46.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:46 smithi002 ceph-mon[28203]: from='mgr.? 172.21.15.99:0/2511772451' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-17T00:12:46.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:46 smithi002 ceph-mon[28203]: from='mgr.? 172.21.15.99:0/2511772451' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-17T00:12:46.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:46 smithi002 ceph-mon[28203]: from='mgr.? 172.21.15.99:0/2511772451' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-17T00:12:46.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:46 smithi099 ceph-mon[24594]: Standby manager daemon x started 2024-09-17T00:12:46.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:46 smithi099 ceph-mon[24594]: from='mgr.? 172.21.15.99:0/2511772451' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-17T00:12:46.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:46 smithi099 ceph-mon[24594]: from='mgr.? 172.21.15.99:0/2511772451' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-17T00:12:46.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:46 smithi099 ceph-mon[24594]: from='mgr.? 172.21.15.99:0/2511772451' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-17T00:12:46.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:46 smithi099 ceph-mon[24594]: from='mgr.? 172.21.15.99:0/2511772451' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-17T00:12:47.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:47 smithi099 ceph-mon[24594]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T00:12:47.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:47 smithi099 ceph-mon[24594]: mgrmap e14: y(active, since 62s), standbys: x 2024-09-17T00:12:47.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:47 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-17T00:12:47.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:47 smithi099 ceph-mon[24594]: from='client.? 172.21.15.2:0/3304454039' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2030f6f2-3659-4b78-bca5-b74d15218c34"}]: dispatch 2024-09-17T00:12:47.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:47 smithi099 ceph-mon[24594]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2030f6f2-3659-4b78-bca5-b74d15218c34"}]: dispatch 2024-09-17T00:12:47.968 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:47 smithi099 ceph-mon[24594]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2030f6f2-3659-4b78-bca5-b74d15218c34"}]': finished 2024-09-17T00:12:47.968 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:47 smithi099 ceph-mon[24594]: osdmap e5: 1 total, 0 up, 1 in 2024-09-17T00:12:47.968 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:47 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T00:12:47.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:47 smithi002 ceph-mon[23737]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T00:12:47.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:47 smithi002 ceph-mon[23737]: mgrmap e14: y(active, since 62s), standbys: x 2024-09-17T00:12:47.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:47 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-17T00:12:47.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:47 smithi002 ceph-mon[23737]: from='client.? 172.21.15.2:0/3304454039' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2030f6f2-3659-4b78-bca5-b74d15218c34"}]: dispatch 2024-09-17T00:12:47.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:47 smithi002 ceph-mon[23737]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2030f6f2-3659-4b78-bca5-b74d15218c34"}]: dispatch 2024-09-17T00:12:47.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:47 smithi002 ceph-mon[23737]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2030f6f2-3659-4b78-bca5-b74d15218c34"}]': finished 2024-09-17T00:12:47.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:47 smithi002 ceph-mon[23737]: osdmap e5: 1 total, 0 up, 1 in 2024-09-17T00:12:47.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:47 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T00:12:47.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:47 smithi002 ceph-mon[28203]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T00:12:47.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:47 smithi002 ceph-mon[28203]: mgrmap e14: y(active, since 62s), standbys: x 2024-09-17T00:12:47.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:47 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-17T00:12:47.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:47 smithi002 ceph-mon[28203]: from='client.? 172.21.15.2:0/3304454039' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2030f6f2-3659-4b78-bca5-b74d15218c34"}]: dispatch 2024-09-17T00:12:47.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:47 smithi002 ceph-mon[28203]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2030f6f2-3659-4b78-bca5-b74d15218c34"}]: dispatch 2024-09-17T00:12:47.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:47 smithi002 ceph-mon[28203]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2030f6f2-3659-4b78-bca5-b74d15218c34"}]': finished 2024-09-17T00:12:47.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:47 smithi002 ceph-mon[28203]: osdmap e5: 1 total, 0 up, 1 in 2024-09-17T00:12:47.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:47 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T00:12:48.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:48 smithi099 ceph-mon[24594]: from='client.? 172.21.15.2:0/2026733189' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-17T00:12:48.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:48 smithi002 ceph-mon[23737]: from='client.? 172.21.15.2:0/2026733189' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-17T00:12:48.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:48 smithi002 ceph-mon[28203]: from='client.? 172.21.15.2:0/2026733189' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-17T00:12:49.900 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:49 smithi002 ceph-mon[23737]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T00:12:49.901 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:49 smithi002 ceph-mon[28203]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T00:12:49.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:49 smithi099 ceph-mon[24594]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T00:12:51.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:51 smithi002 ceph-mon[28203]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T00:12:51.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:51 smithi002 ceph-mon[23737]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T00:12:51.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:51 smithi099 ceph-mon[24594]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T00:12:53.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:53 smithi002 ceph-mon[23737]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T00:12:53.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:53 smithi002 ceph-mon[28203]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T00:12:53.966 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:53 smithi099 ceph-mon[24594]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T00:12:54.923 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:54 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-09-17T00:12:54.923 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:54 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:12:54.923 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:54 smithi002 ceph-mon[23737]: Deploying daemon osd.0 on smithi002 2024-09-17T00:12:54.923 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:54 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-09-17T00:12:54.924 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:54 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:12:54.924 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:54 smithi002 ceph-mon[28203]: Deploying daemon osd.0 on smithi002 2024-09-17T00:12:54.933 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:54 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-09-17T00:12:54.933 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:54 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:12:54.933 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:54 smithi099 ceph-mon[24594]: Deploying daemon osd.0 on smithi002 2024-09-17T00:12:55.853 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:55 smithi002 ceph-mon[28203]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T00:12:55.854 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:55 smithi002 ceph-mon[23737]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T00:12:55.966 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:55 smithi099 ceph-mon[24594]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T00:12:57.869 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:57 smithi002 ceph-mon[23737]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T00:12:57.869 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:57 smithi002 ceph-mon[28203]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T00:12:57.966 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:57 smithi099 ceph-mon[24594]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T00:12:59.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:59 smithi002 ceph-mon[23737]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T00:12:59.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:59 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:12:59.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:59 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T00:12:59.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:59 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:12:59.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:12:59 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:12:59.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:59 smithi002 ceph-mon[28203]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T00:12:59.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:59 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:12:59.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:59 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T00:12:59.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:59 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:12:59.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:12:59 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:13:00.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:59 smithi099 ceph-mon[24594]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T00:13:00.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:59 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:13:00.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:59 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T00:13:00.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:59 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:13:00.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:12:59 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:13:02.137 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:01 smithi002 ceph-mon[23737]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T00:13:02.138 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:01 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:13:02.138 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:01 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:13:02.138 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:01 smithi002 ceph-mon[28203]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T00:13:02.138 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:01 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:13:02.139 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:01 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:13:02.168 INFO:teuthology.orchestra.run.smithi002.stdout:Created osd(s) 0 on host 'smithi002' 2024-09-17T00:13:02.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:01 smithi099 ceph-mon[24594]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T00:13:02.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:01 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:13:02.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:01 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:13:02.782 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:02 smithi002 ceph-mon[23737]: from='osd.0 [v2:172.21.15.2:6802/2656745758,v1:172.21.15.2:6803/2656745758]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-17T00:13:02.783 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:02 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:13:02.783 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:02 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T00:13:02.783 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:02 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:13:02.783 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:02 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:13:02.783 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:02 smithi002 ceph-mon[28203]: from='osd.0 [v2:172.21.15.2:6802/2656745758,v1:172.21.15.2:6803/2656745758]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-17T00:13:02.783 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:02 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:13:02.783 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:02 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T00:13:02.783 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:02 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:13:02.783 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:02 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:13:02.807 DEBUG:teuthology.orchestra.run.smithi002:osd.0> sudo journalctl -f -n 0 -u ceph-3aad27a2-7489-11ef-bceb-c7b262605968@osd.0.service 2024-09-17T00:13:02.810 INFO:tasks.cephadm:Deploying osd.1 on smithi002 with /dev/vg_nvme/lv_3... 2024-09-17T00:13:02.810 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3aad27a2-7489-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_3 2024-09-17T00:13:03.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:02 smithi099 ceph-mon[24594]: from='osd.0 [v2:172.21.15.2:6802/2656745758,v1:172.21.15.2:6803/2656745758]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-17T00:13:03.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:02 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:13:03.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:02 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T00:13:03.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:02 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:13:03.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:02 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:13:03.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:03 smithi002 ceph-mon[23737]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T00:13:03.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:03 smithi002 ceph-mon[23737]: from='osd.0 [v2:172.21.15.2:6802/2656745758,v1:172.21.15.2:6803/2656745758]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-17T00:13:03.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:03 smithi002 ceph-mon[23737]: osdmap e6: 1 total, 0 up, 1 in 2024-09-17T00:13:03.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:03 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T00:13:03.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:03 smithi002 ceph-mon[23737]: from='osd.0 [v2:172.21.15.2:6802/2656745758,v1:172.21.15.2:6803/2656745758]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]: dispatch 2024-09-17T00:13:03.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:03 smithi002 ceph-mon[28203]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T00:13:03.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:03 smithi002 ceph-mon[28203]: from='osd.0 [v2:172.21.15.2:6802/2656745758,v1:172.21.15.2:6803/2656745758]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-17T00:13:03.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:03 smithi002 ceph-mon[28203]: osdmap e6: 1 total, 0 up, 1 in 2024-09-17T00:13:03.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:03 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T00:13:03.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:03 smithi002 ceph-mon[28203]: from='osd.0 [v2:172.21.15.2:6802/2656745758,v1:172.21.15.2:6803/2656745758]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]: dispatch 2024-09-17T00:13:03.980 INFO:journalctl@ceph.osd.0.smithi002.stdout:Sep 17 00:13:03 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-0[31022]: 2024-09-17T00:13:03.744+0000 7f6fceef7700 -1 osd.0 0 waiting for initial osdmap 2024-09-17T00:13:03.981 INFO:journalctl@ceph.osd.0.smithi002.stdout:Sep 17 00:13:03 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-0[31022]: 2024-09-17T00:13:03.746+0000 7f6fc908d700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T00:13:04.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:03 smithi099 ceph-mon[24594]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T00:13:04.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:03 smithi099 ceph-mon[24594]: from='osd.0 [v2:172.21.15.2:6802/2656745758,v1:172.21.15.2:6803/2656745758]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-17T00:13:04.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:03 smithi099 ceph-mon[24594]: osdmap e6: 1 total, 0 up, 1 in 2024-09-17T00:13:04.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:03 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T00:13:04.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:03 smithi099 ceph-mon[24594]: from='osd.0 [v2:172.21.15.2:6802/2656745758,v1:172.21.15.2:6803/2656745758]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]: dispatch 2024-09-17T00:13:05.010 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:04 smithi099 ceph-mon[24594]: from='osd.0 [v2:172.21.15.2:6802/2656745758,v1:172.21.15.2:6803/2656745758]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]': finished 2024-09-17T00:13:05.010 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:04 smithi099 ceph-mon[24594]: osdmap e7: 1 total, 0 up, 1 in 2024-09-17T00:13:05.010 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:04 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T00:13:05.010 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:04 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T00:13:05.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:04 smithi002 ceph-mon[23737]: from='osd.0 [v2:172.21.15.2:6802/2656745758,v1:172.21.15.2:6803/2656745758]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]': finished 2024-09-17T00:13:05.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:04 smithi002 ceph-mon[23737]: osdmap e7: 1 total, 0 up, 1 in 2024-09-17T00:13:05.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:04 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T00:13:05.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:04 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T00:13:05.230 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:04 smithi002 ceph-mon[28203]: from='osd.0 [v2:172.21.15.2:6802/2656745758,v1:172.21.15.2:6803/2656745758]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]': finished 2024-09-17T00:13:05.230 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:04 smithi002 ceph-mon[28203]: osdmap e7: 1 total, 0 up, 1 in 2024-09-17T00:13:05.230 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:04 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T00:13:05.230 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:04 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T00:13:06.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:05 smithi002 ceph-mon[23737]: purged_snaps scrub starts 2024-09-17T00:13:06.118 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:05 smithi002 ceph-mon[23737]: purged_snaps scrub ok 2024-09-17T00:13:06.118 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:05 smithi002 ceph-mon[23737]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T00:13:06.118 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:05 smithi002 ceph-mon[23737]: osd.0 [v2:172.21.15.2:6802/2656745758,v1:172.21.15.2:6803/2656745758] boot 2024-09-17T00:13:06.118 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:05 smithi002 ceph-mon[23737]: osdmap e8: 1 total, 1 up, 1 in 2024-09-17T00:13:06.118 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:05 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T00:13:06.118 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:05 smithi002 ceph-mon[28203]: purged_snaps scrub starts 2024-09-17T00:13:06.118 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:05 smithi002 ceph-mon[28203]: purged_snaps scrub ok 2024-09-17T00:13:06.118 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:05 smithi002 ceph-mon[28203]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T00:13:06.118 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:05 smithi002 ceph-mon[28203]: osd.0 [v2:172.21.15.2:6802/2656745758,v1:172.21.15.2:6803/2656745758] boot 2024-09-17T00:13:06.118 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:05 smithi002 ceph-mon[28203]: osdmap e8: 1 total, 1 up, 1 in 2024-09-17T00:13:06.118 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:05 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T00:13:06.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:05 smithi099 ceph-mon[24594]: purged_snaps scrub starts 2024-09-17T00:13:06.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:05 smithi099 ceph-mon[24594]: purged_snaps scrub ok 2024-09-17T00:13:06.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:05 smithi099 ceph-mon[24594]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-17T00:13:06.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:05 smithi099 ceph-mon[24594]: osd.0 [v2:172.21.15.2:6802/2656745758,v1:172.21.15.2:6803/2656745758] boot 2024-09-17T00:13:06.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:05 smithi099 ceph-mon[24594]: osdmap e8: 1 total, 1 up, 1 in 2024-09-17T00:13:06.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:05 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T00:13:07.058 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:06 smithi002 ceph-mon[23737]: osdmap e9: 1 total, 1 up, 1 in 2024-09-17T00:13:07.059 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:06 smithi002 ceph-mon[28203]: osdmap e9: 1 total, 1 up, 1 in 2024-09-17T00:13:07.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:06 smithi099 ceph-mon[24594]: osdmap e9: 1 total, 1 up, 1 in 2024-09-17T00:13:08.134 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:07 smithi002 ceph-mon[23737]: pgmap v40: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-09-17T00:13:08.134 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:07 smithi002 ceph-mon[23737]: Detected new or changed devices on smithi002 2024-09-17T00:13:08.134 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:07 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:13:08.135 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:07 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-17T00:13:08.135 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:07 smithi002 ceph-mon[23737]: Adjusting osd_memory_target on smithi002 to 16068M 2024-09-17T00:13:08.135 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:07 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:13:08.135 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:07 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:13:08.135 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:07 smithi002 ceph-mon[28203]: pgmap v40: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-09-17T00:13:08.136 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:07 smithi002 ceph-mon[28203]: Detected new or changed devices on smithi002 2024-09-17T00:13:08.136 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:07 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:13:08.136 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:07 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-17T00:13:08.136 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:07 smithi002 ceph-mon[28203]: Adjusting osd_memory_target on smithi002 to 16068M 2024-09-17T00:13:08.136 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:07 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:13:08.136 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:07 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:13:08.141 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-17T00:13:08.160 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3aad27a2-7489-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi002:vg_nvme/lv_3 2024-09-17T00:13:08.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:07 smithi099 ceph-mon[24594]: pgmap v40: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-09-17T00:13:08.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:07 smithi099 ceph-mon[24594]: Detected new or changed devices on smithi002 2024-09-17T00:13:08.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:07 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:13:08.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:07 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-17T00:13:08.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:07 smithi099 ceph-mon[24594]: Adjusting osd_memory_target on smithi002 to 16068M 2024-09-17T00:13:08.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:07 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:13:08.218 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:07 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:13:10.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:09 smithi099 ceph-mon[24594]: pgmap v41: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-09-17T00:13:10.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:09 smithi002 ceph-mon[23737]: pgmap v41: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-09-17T00:13:10.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:09 smithi002 ceph-mon[28203]: pgmap v41: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-09-17T00:13:10.904 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:10 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-17T00:13:10.904 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:10 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-17T00:13:10.904 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:10 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:13:10.904 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:10 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-17T00:13:10.904 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:10 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-17T00:13:10.905 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:10 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:13:11.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:10 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-17T00:13:11.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:10 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-17T00:13:11.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:10 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:13:12.216 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:11 smithi099 ceph-mon[24594]: pgmap v42: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-09-17T00:13:12.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:11 smithi099 ceph-mon[24594]: from='client.14250 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi002:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:13:12.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:11 smithi002 ceph-mon[23737]: pgmap v42: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-09-17T00:13:12.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:11 smithi002 ceph-mon[23737]: from='client.14250 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi002:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:13:12.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:11 smithi002 ceph-mon[28203]: pgmap v42: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-09-17T00:13:12.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:11 smithi002 ceph-mon[28203]: from='client.14250 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi002:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:13:13.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:13 smithi099 ceph-mon[24594]: pgmap v43: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-09-17T00:13:13.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:13 smithi099 ceph-mon[24594]: from='client.? 172.21.15.2:0/3275571347' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e3883c47-fd8d-444f-8057-25d08b59af31"}]: dispatch 2024-09-17T00:13:13.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:13 smithi099 ceph-mon[24594]: from='client.? 172.21.15.2:0/3275571347' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e3883c47-fd8d-444f-8057-25d08b59af31"}]': finished 2024-09-17T00:13:13.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:13 smithi099 ceph-mon[24594]: osdmap e10: 2 total, 1 up, 2 in 2024-09-17T00:13:13.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:13 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T00:13:13.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:13 smithi099 ceph-mon[24594]: from='client.? 172.21.15.2:0/3620736571' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-17T00:13:13.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:13 smithi002 ceph-mon[23737]: pgmap v43: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-09-17T00:13:13.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:13 smithi002 ceph-mon[23737]: from='client.? 172.21.15.2:0/3275571347' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e3883c47-fd8d-444f-8057-25d08b59af31"}]: dispatch 2024-09-17T00:13:13.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:13 smithi002 ceph-mon[23737]: from='client.? 172.21.15.2:0/3275571347' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e3883c47-fd8d-444f-8057-25d08b59af31"}]': finished 2024-09-17T00:13:13.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:13 smithi002 ceph-mon[23737]: osdmap e10: 2 total, 1 up, 2 in 2024-09-17T00:13:13.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:13 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T00:13:13.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:13 smithi002 ceph-mon[23737]: from='client.? 172.21.15.2:0/3620736571' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-17T00:13:13.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:13 smithi002 ceph-mon[28203]: pgmap v43: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-09-17T00:13:13.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:13 smithi002 ceph-mon[28203]: from='client.? 172.21.15.2:0/3275571347' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e3883c47-fd8d-444f-8057-25d08b59af31"}]: dispatch 2024-09-17T00:13:13.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:13 smithi002 ceph-mon[28203]: from='client.? 172.21.15.2:0/3275571347' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e3883c47-fd8d-444f-8057-25d08b59af31"}]': finished 2024-09-17T00:13:13.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:13 smithi002 ceph-mon[28203]: osdmap e10: 2 total, 1 up, 2 in 2024-09-17T00:13:13.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:13 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T00:13:13.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:13 smithi002 ceph-mon[28203]: from='client.? 172.21.15.2:0/3620736571' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-17T00:13:15.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:15 smithi002 ceph-mon[23737]: pgmap v45: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-09-17T00:13:15.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:15 smithi002 ceph-mon[28203]: pgmap v45: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-09-17T00:13:15.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:15 smithi099 ceph-mon[24594]: pgmap v45: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-09-17T00:13:17.604 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:17 smithi002 ceph-mon[23737]: pgmap v46: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-09-17T00:13:17.604 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:17 smithi002 ceph-mon[28203]: pgmap v46: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-09-17T00:13:17.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:17 smithi099 ceph-mon[24594]: pgmap v46: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-09-17T00:13:19.629 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:19 smithi002 ceph-mon[28203]: pgmap v47: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-09-17T00:13:19.629 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:19 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-09-17T00:13:19.629 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:19 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:13:19.629 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:19 smithi002 ceph-mon[23737]: pgmap v47: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-09-17T00:13:19.629 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:19 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-09-17T00:13:19.629 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:19 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:13:19.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:19 smithi099 ceph-mon[24594]: pgmap v47: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-09-17T00:13:19.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:19 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-09-17T00:13:19.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:19 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:13:20.609 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:20 smithi002 ceph-mon[28203]: Deploying daemon osd.1 on smithi002 2024-09-17T00:13:20.609 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:20 smithi002 ceph-mon[23737]: Deploying daemon osd.1 on smithi002 2024-09-17T00:13:20.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:20 smithi099 ceph-mon[24594]: Deploying daemon osd.1 on smithi002 2024-09-17T00:13:21.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:21 smithi002 ceph-mon[23737]: pgmap v48: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-09-17T00:13:21.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:21 smithi002 ceph-mon[28203]: pgmap v48: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-09-17T00:13:21.966 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:21 smithi099 ceph-mon[24594]: pgmap v48: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-09-17T00:13:23.848 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:23 smithi002 ceph-mon[23737]: pgmap v49: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-09-17T00:13:23.848 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:23 smithi002 ceph-mon[28203]: pgmap v49: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-09-17T00:13:23.966 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:23 smithi099 ceph-mon[24594]: pgmap v49: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-09-17T00:13:25.278 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:24 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:13:25.278 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:24 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T00:13:25.278 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:24 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:13:25.279 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:24 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:13:25.279 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:24 smithi002 ceph-mon[28203]: pgmap v50: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-09-17T00:13:25.279 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:24 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:13:25.279 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:24 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T00:13:25.279 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:24 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:13:25.279 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:24 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:13:25.279 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:24 smithi002 ceph-mon[23737]: pgmap v50: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-09-17T00:13:25.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:24 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:13:25.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:24 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T00:13:25.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:24 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:13:25.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:24 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:13:25.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:24 smithi099 ceph-mon[24594]: pgmap v50: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-09-17T00:13:26.709 INFO:teuthology.orchestra.run.smithi002.stdout:Created osd(s) 1 on host 'smithi002' 2024-09-17T00:13:27.566 DEBUG:teuthology.orchestra.run.smithi002:osd.1> sudo journalctl -f -n 0 -u ceph-3aad27a2-7489-11ef-bceb-c7b262605968@osd.1.service 2024-09-17T00:13:27.568 INFO:tasks.cephadm:Deploying osd.2 on smithi002 with /dev/vg_nvme/lv_2... 2024-09-17T00:13:27.568 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3aad27a2-7489-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_2 2024-09-17T00:13:27.805 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:27 smithi002 ceph-mon[23737]: pgmap v51: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-09-17T00:13:27.805 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:27 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:13:27.805 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:27 smithi002 ceph-mon[23737]: from='osd.1 [v2:172.21.15.2:6810/981024217,v1:172.21.15.2:6811/981024217]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-17T00:13:27.806 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:27 smithi002 ceph-mon[28203]: pgmap v51: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-09-17T00:13:27.806 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:27 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:13:27.806 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:27 smithi002 ceph-mon[28203]: from='osd.1 [v2:172.21.15.2:6810/981024217,v1:172.21.15.2:6811/981024217]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-17T00:13:27.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:27 smithi099 ceph-mon[24594]: pgmap v51: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-09-17T00:13:27.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:27 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:13:27.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:27 smithi099 ceph-mon[24594]: from='osd.1 [v2:172.21.15.2:6810/981024217,v1:172.21.15.2:6811/981024217]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-17T00:13:28.780 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:28 smithi002 ceph-mon[28203]: from='osd.1 [v2:172.21.15.2:6810/981024217,v1:172.21.15.2:6811/981024217]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-17T00:13:28.780 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:28 smithi002 ceph-mon[28203]: osdmap e11: 2 total, 1 up, 2 in 2024-09-17T00:13:28.780 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:28 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T00:13:28.780 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:28 smithi002 ceph-mon[28203]: from='osd.1 [v2:172.21.15.2:6810/981024217,v1:172.21.15.2:6811/981024217]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]: dispatch 2024-09-17T00:13:28.780 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:28 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:13:28.781 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:28 smithi002 ceph-mon[23737]: from='osd.1 [v2:172.21.15.2:6810/981024217,v1:172.21.15.2:6811/981024217]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-17T00:13:28.781 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:28 smithi002 ceph-mon[23737]: osdmap e11: 2 total, 1 up, 2 in 2024-09-17T00:13:28.781 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:28 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T00:13:28.781 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:28 smithi002 ceph-mon[23737]: from='osd.1 [v2:172.21.15.2:6810/981024217,v1:172.21.15.2:6811/981024217]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]: dispatch 2024-09-17T00:13:28.781 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:28 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:13:28.781 INFO:journalctl@ceph.osd.1.smithi002.stdout:Sep 17 00:13:28 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-1[33959]: 2024-09-17T00:13:28.723+0000 7f90f35b5700 -1 osd.1 0 waiting for initial osdmap 2024-09-17T00:13:28.781 INFO:journalctl@ceph.osd.1.smithi002.stdout:Sep 17 00:13:28 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-1[33959]: 2024-09-17T00:13:28.726+0000 7f90ef74f700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T00:13:29.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:28 smithi099 ceph-mon[24594]: from='osd.1 [v2:172.21.15.2:6810/981024217,v1:172.21.15.2:6811/981024217]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-17T00:13:29.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:28 smithi099 ceph-mon[24594]: osdmap e11: 2 total, 1 up, 2 in 2024-09-17T00:13:29.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:28 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T00:13:29.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:28 smithi099 ceph-mon[24594]: from='osd.1 [v2:172.21.15.2:6810/981024217,v1:172.21.15.2:6811/981024217]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]: dispatch 2024-09-17T00:13:29.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:28 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:13:29.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:29 smithi002 ceph-mon[28203]: pgmap v53: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-09-17T00:13:29.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:29 smithi002 ceph-mon[28203]: from='osd.1 [v2:172.21.15.2:6810/981024217,v1:172.21.15.2:6811/981024217]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]': finished 2024-09-17T00:13:29.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:29 smithi002 ceph-mon[28203]: osdmap e12: 2 total, 1 up, 2 in 2024-09-17T00:13:29.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:29 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T00:13:29.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:29 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T00:13:29.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:29 smithi002 ceph-mon[23737]: pgmap v53: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-09-17T00:13:29.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:29 smithi002 ceph-mon[23737]: from='osd.1 [v2:172.21.15.2:6810/981024217,v1:172.21.15.2:6811/981024217]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]': finished 2024-09-17T00:13:29.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:29 smithi002 ceph-mon[23737]: osdmap e12: 2 total, 1 up, 2 in 2024-09-17T00:13:29.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:29 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T00:13:29.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:29 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T00:13:30.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:29 smithi099 ceph-mon[24594]: pgmap v53: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-09-17T00:13:30.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:29 smithi099 ceph-mon[24594]: from='osd.1 [v2:172.21.15.2:6810/981024217,v1:172.21.15.2:6811/981024217]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]': finished 2024-09-17T00:13:30.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:29 smithi099 ceph-mon[24594]: osdmap e12: 2 total, 1 up, 2 in 2024-09-17T00:13:30.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:29 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T00:13:30.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:29 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T00:13:30.713 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-17T00:13:30.739 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3aad27a2-7489-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi002:vg_nvme/lv_2 2024-09-17T00:13:30.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:30 smithi002 ceph-mon[23737]: purged_snaps scrub starts 2024-09-17T00:13:30.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:30 smithi002 ceph-mon[23737]: purged_snaps scrub ok 2024-09-17T00:13:30.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:30 smithi002 ceph-mon[23737]: osd.1 [v2:172.21.15.2:6810/981024217,v1:172.21.15.2:6811/981024217] boot 2024-09-17T00:13:30.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:30 smithi002 ceph-mon[23737]: osdmap e13: 2 total, 2 up, 2 in 2024-09-17T00:13:30.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:30 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T00:13:30.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:30 smithi002 ceph-mon[28203]: purged_snaps scrub starts 2024-09-17T00:13:30.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:30 smithi002 ceph-mon[28203]: purged_snaps scrub ok 2024-09-17T00:13:30.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:30 smithi002 ceph-mon[28203]: osd.1 [v2:172.21.15.2:6810/981024217,v1:172.21.15.2:6811/981024217] boot 2024-09-17T00:13:30.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:30 smithi002 ceph-mon[28203]: osdmap e13: 2 total, 2 up, 2 in 2024-09-17T00:13:30.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:30 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T00:13:31.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:30 smithi099 ceph-mon[24594]: purged_snaps scrub starts 2024-09-17T00:13:31.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:30 smithi099 ceph-mon[24594]: purged_snaps scrub ok 2024-09-17T00:13:31.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:30 smithi099 ceph-mon[24594]: osd.1 [v2:172.21.15.2:6810/981024217,v1:172.21.15.2:6811/981024217] boot 2024-09-17T00:13:31.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:30 smithi099 ceph-mon[24594]: osdmap e13: 2 total, 2 up, 2 in 2024-09-17T00:13:31.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:30 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T00:13:31.956 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:31 smithi002 ceph-mon[23737]: pgmap v56: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-09-17T00:13:31.956 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:31 smithi002 ceph-mon[23737]: osdmap e14: 2 total, 2 up, 2 in 2024-09-17T00:13:31.957 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:31 smithi002 ceph-mon[28203]: pgmap v56: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-09-17T00:13:31.957 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:31 smithi002 ceph-mon[28203]: osdmap e14: 2 total, 2 up, 2 in 2024-09-17T00:13:32.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:31 smithi099 ceph-mon[24594]: pgmap v56: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-09-17T00:13:32.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:31 smithi099 ceph-mon[24594]: osdmap e14: 2 total, 2 up, 2 in 2024-09-17T00:13:34.068 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:33 smithi002 ceph-mon[23737]: pgmap v58: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-09-17T00:13:34.068 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:33 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:13:34.069 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:33 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-17T00:13:34.069 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:33 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-17T00:13:34.069 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:33 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:13:34.069 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:33 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:13:34.069 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:33 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T00:13:34.069 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:33 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:13:34.069 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:33 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:13:34.069 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:33 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:13:34.069 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:33 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-17T00:13:34.069 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:33 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-17T00:13:34.069 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:33 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:13:34.070 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:33 smithi002 ceph-mon[28203]: pgmap v58: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-09-17T00:13:34.070 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:33 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:13:34.070 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:33 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-17T00:13:34.070 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:33 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-17T00:13:34.070 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:33 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:13:34.070 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:33 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:13:34.070 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:33 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T00:13:34.070 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:33 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:13:34.070 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:33 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:13:34.070 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:33 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:13:34.070 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:33 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-17T00:13:34.070 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:33 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-17T00:13:34.071 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:33 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:13:34.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:33 smithi099 ceph-mon[24594]: pgmap v58: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-09-17T00:13:34.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:33 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:13:34.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:33 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-17T00:13:34.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:33 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-17T00:13:34.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:33 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:13:34.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:33 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:13:34.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:33 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T00:13:34.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:33 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:13:34.218 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:33 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:13:34.218 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:33 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:13:34.218 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:33 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-17T00:13:34.218 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:33 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-17T00:13:34.218 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:33 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:13:35.063 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:34 smithi002 ceph-mon[28203]: Detected new or changed devices on smithi002 2024-09-17T00:13:35.063 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:34 smithi002 ceph-mon[28203]: Adjusting osd_memory_target on smithi002 to 8034M 2024-09-17T00:13:35.063 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:34 smithi002 ceph-mon[28203]: from='client.14277 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi002:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:13:35.063 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:34 smithi002 ceph-mon[23737]: Detected new or changed devices on smithi002 2024-09-17T00:13:35.063 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:34 smithi002 ceph-mon[23737]: Adjusting osd_memory_target on smithi002 to 8034M 2024-09-17T00:13:35.063 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:34 smithi002 ceph-mon[23737]: from='client.14277 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi002:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:13:35.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:34 smithi099 ceph-mon[24594]: Detected new or changed devices on smithi002 2024-09-17T00:13:35.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:34 smithi099 ceph-mon[24594]: Adjusting osd_memory_target on smithi002 to 8034M 2024-09-17T00:13:35.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:34 smithi099 ceph-mon[24594]: from='client.14277 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi002:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:13:36.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:35 smithi099 ceph-mon[24594]: pgmap v59: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-09-17T00:13:36.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:35 smithi002 ceph-mon[23737]: pgmap v59: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-09-17T00:13:36.230 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:35 smithi002 ceph-mon[28203]: pgmap v59: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-09-17T00:13:37.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:36 smithi099 ceph-mon[24594]: from='client.? 172.21.15.2:0/2076699889' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9d042533-63e5-4f38-a8db-10b61e9b39d9"}]: dispatch 2024-09-17T00:13:37.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:36 smithi099 ceph-mon[24594]: from='client.? 172.21.15.2:0/2076699889' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9d042533-63e5-4f38-a8db-10b61e9b39d9"}]': finished 2024-09-17T00:13:37.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:36 smithi099 ceph-mon[24594]: osdmap e15: 3 total, 2 up, 3 in 2024-09-17T00:13:37.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:36 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T00:13:37.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:36 smithi099 ceph-mon[24594]: from='client.? 172.21.15.2:0/2438657616' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-17T00:13:37.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:36 smithi002 ceph-mon[23737]: from='client.? 172.21.15.2:0/2076699889' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9d042533-63e5-4f38-a8db-10b61e9b39d9"}]: dispatch 2024-09-17T00:13:37.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:36 smithi002 ceph-mon[23737]: from='client.? 172.21.15.2:0/2076699889' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9d042533-63e5-4f38-a8db-10b61e9b39d9"}]': finished 2024-09-17T00:13:37.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:36 smithi002 ceph-mon[23737]: osdmap e15: 3 total, 2 up, 3 in 2024-09-17T00:13:37.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:36 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T00:13:37.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:36 smithi002 ceph-mon[23737]: from='client.? 172.21.15.2:0/2438657616' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-17T00:13:37.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:36 smithi002 ceph-mon[28203]: from='client.? 172.21.15.2:0/2076699889' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9d042533-63e5-4f38-a8db-10b61e9b39d9"}]: dispatch 2024-09-17T00:13:37.230 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:36 smithi002 ceph-mon[28203]: from='client.? 172.21.15.2:0/2076699889' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9d042533-63e5-4f38-a8db-10b61e9b39d9"}]': finished 2024-09-17T00:13:37.230 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:36 smithi002 ceph-mon[28203]: osdmap e15: 3 total, 2 up, 3 in 2024-09-17T00:13:37.230 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:36 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T00:13:37.230 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:36 smithi002 ceph-mon[28203]: from='client.? 172.21.15.2:0/2438657616' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-17T00:13:37.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:37 smithi002 ceph-mon[23737]: pgmap v61: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-09-17T00:13:37.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:37 smithi002 ceph-mon[28203]: pgmap v61: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-09-17T00:13:38.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:37 smithi099 ceph-mon[24594]: pgmap v61: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-09-17T00:13:40.031 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:39 smithi002 ceph-mon[23737]: pgmap v62: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-09-17T00:13:40.031 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:39 smithi002 ceph-mon[28203]: pgmap v62: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-09-17T00:13:40.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:39 smithi099 ceph-mon[24594]: pgmap v62: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-09-17T00:13:41.881 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:41 smithi002 ceph-mon[28203]: pgmap v63: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-09-17T00:13:41.881 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:41 smithi002 ceph-mon[23737]: pgmap v63: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-09-17T00:13:42.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:41 smithi099 ceph-mon[24594]: pgmap v63: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-09-17T00:13:42.901 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:42 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-09-17T00:13:42.902 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:42 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:13:42.902 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:42 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-09-17T00:13:42.902 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:42 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:13:43.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:42 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-09-17T00:13:43.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:42 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:13:43.961 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:43 smithi002 ceph-mon[28203]: pgmap v64: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-09-17T00:13:43.961 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:43 smithi002 ceph-mon[28203]: Deploying daemon osd.2 on smithi002 2024-09-17T00:13:43.961 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:43 smithi002 ceph-mon[23737]: pgmap v64: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-09-17T00:13:43.961 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:43 smithi002 ceph-mon[23737]: Deploying daemon osd.2 on smithi002 2024-09-17T00:13:44.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:43 smithi099 ceph-mon[24594]: pgmap v64: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-09-17T00:13:44.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:43 smithi099 ceph-mon[24594]: Deploying daemon osd.2 on smithi002 2024-09-17T00:13:44.944 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:44 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T00:13:44.944 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:44 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T00:13:44.944 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:44 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T00:13:44.945 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:44 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T00:13:45.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:44 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T00:13:45.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:44 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T00:13:45.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:45 smithi002 ceph-mon[23737]: pgmap v65: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-09-17T00:13:45.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:45 smithi002 ceph-mon[28203]: pgmap v65: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-09-17T00:13:46.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:45 smithi099 ceph-mon[24594]: pgmap v65: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-09-17T00:13:47.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:47 smithi002 ceph-mon[23737]: pgmap v66: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-09-17T00:13:47.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:47 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:13:47.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:47 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T00:13:47.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:47 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:13:47.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:47 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:13:47.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:47 smithi002 ceph-mon[28203]: pgmap v66: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-09-17T00:13:47.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:47 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:13:47.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:47 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T00:13:47.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:47 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:13:47.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:47 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:13:48.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:47 smithi099 ceph-mon[24594]: pgmap v66: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-09-17T00:13:48.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:47 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:13:48.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:47 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T00:13:48.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:47 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:13:48.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:47 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:13:50.202 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:49 smithi002 ceph-mon[28203]: pgmap v67: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-09-17T00:13:50.203 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:49 smithi002 ceph-mon[23737]: pgmap v67: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-09-17T00:13:50.216 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:49 smithi099 ceph-mon[24594]: pgmap v67: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-09-17T00:13:50.521 INFO:teuthology.orchestra.run.smithi002.stdout:Created osd(s) 2 on host 'smithi002' 2024-09-17T00:13:51.164 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:50 smithi002 ceph-mon[28203]: from='osd.2 [v2:172.21.15.2:6818/2219840596,v1:172.21.15.2:6819/2219840596]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-17T00:13:51.164 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:50 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:13:51.164 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:50 smithi002 ceph-mon[23737]: from='osd.2 [v2:172.21.15.2:6818/2219840596,v1:172.21.15.2:6819/2219840596]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-17T00:13:51.165 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:50 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:13:51.188 DEBUG:teuthology.orchestra.run.smithi002:osd.2> sudo journalctl -f -n 0 -u ceph-3aad27a2-7489-11ef-bceb-c7b262605968@osd.2.service 2024-09-17T00:13:51.191 INFO:tasks.cephadm:Deploying osd.3 on smithi002 with /dev/vg_nvme/lv_1... 2024-09-17T00:13:51.192 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3aad27a2-7489-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_1 2024-09-17T00:13:51.216 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:50 smithi099 ceph-mon[24594]: from='osd.2 [v2:172.21.15.2:6818/2219840596,v1:172.21.15.2:6819/2219840596]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-17T00:13:51.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:50 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:13:52.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:51 smithi099 ceph-mon[24594]: pgmap v68: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-09-17T00:13:52.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:51 smithi099 ceph-mon[24594]: from='osd.2 [v2:172.21.15.2:6818/2219840596,v1:172.21.15.2:6819/2219840596]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-17T00:13:52.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:51 smithi099 ceph-mon[24594]: osdmap e16: 3 total, 2 up, 3 in 2024-09-17T00:13:52.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:51 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T00:13:52.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:51 smithi099 ceph-mon[24594]: from='osd.2 [v2:172.21.15.2:6818/2219840596,v1:172.21.15.2:6819/2219840596]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]: dispatch 2024-09-17T00:13:52.218 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:51 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:13:52.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:51 smithi002 ceph-mon[23737]: pgmap v68: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-09-17T00:13:52.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:51 smithi002 ceph-mon[23737]: from='osd.2 [v2:172.21.15.2:6818/2219840596,v1:172.21.15.2:6819/2219840596]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-17T00:13:52.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:51 smithi002 ceph-mon[23737]: osdmap e16: 3 total, 2 up, 3 in 2024-09-17T00:13:52.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:51 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T00:13:52.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:51 smithi002 ceph-mon[23737]: from='osd.2 [v2:172.21.15.2:6818/2219840596,v1:172.21.15.2:6819/2219840596]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]: dispatch 2024-09-17T00:13:52.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:51 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:13:52.230 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:51 smithi002 ceph-mon[28203]: pgmap v68: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-09-17T00:13:52.230 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:51 smithi002 ceph-mon[28203]: from='osd.2 [v2:172.21.15.2:6818/2219840596,v1:172.21.15.2:6819/2219840596]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-17T00:13:52.230 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:51 smithi002 ceph-mon[28203]: osdmap e16: 3 total, 2 up, 3 in 2024-09-17T00:13:52.230 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:51 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T00:13:52.230 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:51 smithi002 ceph-mon[28203]: from='osd.2 [v2:172.21.15.2:6818/2219840596,v1:172.21.15.2:6819/2219840596]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]: dispatch 2024-09-17T00:13:52.230 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:51 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:13:52.231 INFO:journalctl@ceph.osd.2.smithi002.stdout:Sep 17 00:13:51 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-2[36852]: 2024-09-17T00:13:51.775+0000 7fea9d3c5700 -1 osd.2 0 waiting for initial osdmap 2024-09-17T00:13:52.231 INFO:journalctl@ceph.osd.2.smithi002.stdout:Sep 17 00:13:51 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-2[36852]: 2024-09-17T00:13:51.779+0000 7fea95d58700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T00:13:53.032 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:52 smithi002 ceph-mon[28203]: from='osd.2 [v2:172.21.15.2:6818/2219840596,v1:172.21.15.2:6819/2219840596]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]': finished 2024-09-17T00:13:53.032 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:52 smithi002 ceph-mon[28203]: osdmap e17: 3 total, 2 up, 3 in 2024-09-17T00:13:53.032 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:52 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T00:13:53.032 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:52 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T00:13:53.032 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:52 smithi002 ceph-mon[23737]: from='osd.2 [v2:172.21.15.2:6818/2219840596,v1:172.21.15.2:6819/2219840596]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]': finished 2024-09-17T00:13:53.032 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:52 smithi002 ceph-mon[23737]: osdmap e17: 3 total, 2 up, 3 in 2024-09-17T00:13:53.032 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:52 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T00:13:53.033 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:52 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T00:13:53.118 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:52 smithi099 ceph-mon[24594]: from='osd.2 [v2:172.21.15.2:6818/2219840596,v1:172.21.15.2:6819/2219840596]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]': finished 2024-09-17T00:13:53.118 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:52 smithi099 ceph-mon[24594]: osdmap e17: 3 total, 2 up, 3 in 2024-09-17T00:13:53.118 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:52 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T00:13:53.118 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:52 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T00:13:53.955 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:53 smithi002 ceph-mon[28203]: purged_snaps scrub starts 2024-09-17T00:13:53.955 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:53 smithi002 ceph-mon[28203]: purged_snaps scrub ok 2024-09-17T00:13:53.955 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:53 smithi002 ceph-mon[28203]: pgmap v71: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-09-17T00:13:53.955 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:53 smithi002 ceph-mon[28203]: osd.2 [v2:172.21.15.2:6818/2219840596,v1:172.21.15.2:6819/2219840596] boot 2024-09-17T00:13:53.955 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:53 smithi002 ceph-mon[28203]: osdmap e18: 3 total, 3 up, 3 in 2024-09-17T00:13:53.955 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:53 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T00:13:53.956 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:53 smithi002 ceph-mon[23737]: purged_snaps scrub starts 2024-09-17T00:13:53.956 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:53 smithi002 ceph-mon[23737]: purged_snaps scrub ok 2024-09-17T00:13:53.956 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:53 smithi002 ceph-mon[23737]: pgmap v71: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-09-17T00:13:53.956 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:53 smithi002 ceph-mon[23737]: osd.2 [v2:172.21.15.2:6818/2219840596,v1:172.21.15.2:6819/2219840596] boot 2024-09-17T00:13:53.956 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:53 smithi002 ceph-mon[23737]: osdmap e18: 3 total, 3 up, 3 in 2024-09-17T00:13:53.956 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:53 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T00:13:54.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:53 smithi099 ceph-mon[24594]: purged_snaps scrub starts 2024-09-17T00:13:54.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:53 smithi099 ceph-mon[24594]: purged_snaps scrub ok 2024-09-17T00:13:54.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:53 smithi099 ceph-mon[24594]: pgmap v71: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-09-17T00:13:54.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:53 smithi099 ceph-mon[24594]: osd.2 [v2:172.21.15.2:6818/2219840596,v1:172.21.15.2:6819/2219840596] boot 2024-09-17T00:13:54.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:53 smithi099 ceph-mon[24594]: osdmap e18: 3 total, 3 up, 3 in 2024-09-17T00:13:54.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:53 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T00:13:54.236 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-17T00:13:54.257 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3aad27a2-7489-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi002:vg_nvme/lv_1 2024-09-17T00:13:54.828 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:54 smithi002 ceph-mon[23737]: osdmap e19: 3 total, 3 up, 3 in 2024-09-17T00:13:54.828 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:54 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-09-17T00:13:54.828 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:54 smithi002 ceph-mon[28203]: osdmap e19: 3 total, 3 up, 3 in 2024-09-17T00:13:54.828 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:54 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-09-17T00:13:55.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:54 smithi099 ceph-mon[24594]: osdmap e19: 3 total, 3 up, 3 in 2024-09-17T00:13:55.245 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:54 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-09-17T00:13:55.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:55 smithi002 ceph-mon[23737]: pgmap v74: 0 pgs: ; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-09-17T00:13:55.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:55 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-09-17T00:13:55.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:55 smithi002 ceph-mon[23737]: osdmap e20: 3 total, 3 up, 3 in 2024-09-17T00:13:55.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:55 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-09-17T00:13:55.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:55 smithi002 ceph-mon[28203]: pgmap v74: 0 pgs: ; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-09-17T00:13:55.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:55 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-09-17T00:13:55.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:55 smithi002 ceph-mon[28203]: osdmap e20: 3 total, 3 up, 3 in 2024-09-17T00:13:55.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:55 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-09-17T00:13:56.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:55 smithi099 ceph-mon[24594]: pgmap v74: 0 pgs: ; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-09-17T00:13:56.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:55 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-09-17T00:13:56.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:55 smithi099 ceph-mon[24594]: osdmap e20: 3 total, 3 up, 3 in 2024-09-17T00:13:56.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:55 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-09-17T00:13:57.116 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:56 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-17T00:13:57.116 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:56 smithi002 ceph-mon[28203]: osdmap e21: 3 total, 3 up, 3 in 2024-09-17T00:13:57.116 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:56 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:13:57.116 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:56 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-17T00:13:57.116 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:56 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-17T00:13:57.116 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:56 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-17T00:13:57.116 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:56 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:13:57.116 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:56 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:13:57.116 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:56 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T00:13:57.116 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:56 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:13:57.116 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:56 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:13:57.117 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:56 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:13:57.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:56 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-17T00:13:57.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:56 smithi002 ceph-mon[23737]: osdmap e21: 3 total, 3 up, 3 in 2024-09-17T00:13:57.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:56 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:13:57.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:56 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-17T00:13:57.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:56 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-17T00:13:57.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:56 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-17T00:13:57.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:56 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:13:57.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:56 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:13:57.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:56 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T00:13:57.117 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:56 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:13:57.118 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:56 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:13:57.118 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:56 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:13:57.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:56 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-17T00:13:57.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:56 smithi099 ceph-mon[24594]: osdmap e21: 3 total, 3 up, 3 in 2024-09-17T00:13:57.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:56 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:13:57.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:56 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-17T00:13:57.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:56 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-17T00:13:57.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:56 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-17T00:13:57.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:56 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:13:57.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:56 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:13:57.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:56 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T00:13:57.218 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:56 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:13:57.218 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:56 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:13:57.218 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:56 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:13:57.390 INFO:journalctl@ceph.osd.0.smithi002.stdout:Sep 17 00:13:57 smithi002 sudo[38152]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-17T00:13:57.390 INFO:journalctl@ceph.osd.0.smithi002.stdout:Sep 17 00:13:57 smithi002 sudo[38152]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-09-17T00:13:57.390 INFO:journalctl@ceph.osd.0.smithi002.stdout:Sep 17 00:13:57 smithi002 sudo[38152]: pam_unix(sudo:session): session closed for user root 2024-09-17T00:13:57.729 INFO:journalctl@ceph.osd.0.smithi002.stdout:Sep 17 00:13:57 smithi002 sudo[38182]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-17T00:13:57.729 INFO:journalctl@ceph.osd.0.smithi002.stdout:Sep 17 00:13:57 smithi002 sudo[38182]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-09-17T00:13:57.729 INFO:journalctl@ceph.osd.0.smithi002.stdout:Sep 17 00:13:57 smithi002 sudo[38182]: pam_unix(sudo:session): session closed for user root 2024-09-17T00:13:58.074 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:57 smithi002 ceph-mon[28203]: pgmap v77: 1 pgs: 1 creating+peering; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-09-17T00:13:58.074 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:57 smithi002 ceph-mon[28203]: Detected new or changed devices on smithi002 2024-09-17T00:13:58.074 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:57 smithi002 ceph-mon[28203]: Adjusting osd_memory_target on smithi002 to 5356M 2024-09-17T00:13:58.074 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:57 smithi002 ceph-mon[28203]: osdmap e22: 3 total, 3 up, 3 in 2024-09-17T00:13:58.074 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:57 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:13:58.074 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:57 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-17T00:13:58.074 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:57 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-17T00:13:58.074 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:57 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:13:58.075 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:57 smithi002 ceph-mon[23737]: pgmap v77: 1 pgs: 1 creating+peering; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-09-17T00:13:58.075 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:57 smithi002 ceph-mon[23737]: Detected new or changed devices on smithi002 2024-09-17T00:13:58.075 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:57 smithi002 ceph-mon[23737]: Adjusting osd_memory_target on smithi002 to 5356M 2024-09-17T00:13:58.075 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:57 smithi002 ceph-mon[23737]: osdmap e22: 3 total, 3 up, 3 in 2024-09-17T00:13:58.075 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:57 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:13:58.075 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:57 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-17T00:13:58.075 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:57 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-17T00:13:58.075 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:57 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:13:58.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:57 smithi099 ceph-mon[24594]: pgmap v77: 1 pgs: 1 creating+peering; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-09-17T00:13:58.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:57 smithi099 ceph-mon[24594]: Detected new or changed devices on smithi002 2024-09-17T00:13:58.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:57 smithi099 ceph-mon[24594]: Adjusting osd_memory_target on smithi002 to 5356M 2024-09-17T00:13:58.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:57 smithi099 ceph-mon[24594]: osdmap e22: 3 total, 3 up, 3 in 2024-09-17T00:13:58.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:57 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:13:58.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:57 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-17T00:13:58.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:57 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-17T00:13:58.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:57 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:13:58.409 INFO:journalctl@ceph.osd.1.smithi002.stdout:Sep 17 00:13:58 smithi002 sudo[38262]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-17T00:13:58.409 INFO:journalctl@ceph.osd.1.smithi002.stdout:Sep 17 00:13:58 smithi002 sudo[38262]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-09-17T00:13:58.409 INFO:journalctl@ceph.osd.1.smithi002.stdout:Sep 17 00:13:58 smithi002 sudo[38262]: pam_unix(sudo:session): session closed for user root 2024-09-17T00:13:58.729 INFO:journalctl@ceph.osd.1.smithi002.stdout:Sep 17 00:13:58 smithi002 sudo[38283]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-17T00:13:58.729 INFO:journalctl@ceph.osd.1.smithi002.stdout:Sep 17 00:13:58 smithi002 sudo[38283]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-09-17T00:13:58.729 INFO:journalctl@ceph.osd.1.smithi002.stdout:Sep 17 00:13:58 smithi002 sudo[38283]: pam_unix(sudo:session): session closed for user root 2024-09-17T00:13:59.074 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:58 smithi002 ceph-mon[23737]: from='client.14310 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi002:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:13:59.074 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:13:58 smithi002 ceph-mon[28203]: from='client.14310 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi002:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:13:59.074 INFO:journalctl@ceph.osd.2.smithi002.stdout:Sep 17 00:13:59 smithi002 sudo[38308]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-17T00:13:59.074 INFO:journalctl@ceph.osd.2.smithi002.stdout:Sep 17 00:13:59 smithi002 sudo[38308]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-09-17T00:13:59.075 INFO:journalctl@ceph.osd.2.smithi002.stdout:Sep 17 00:13:59 smithi002 sudo[38308]: pam_unix(sudo:session): session closed for user root 2024-09-17T00:13:59.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:13:58 smithi099 ceph-mon[24594]: from='client.14310 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi002:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:13:59.729 INFO:journalctl@ceph.osd.2.smithi002.stdout:Sep 17 00:13:59 smithi002 sudo[38318]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-17T00:13:59.729 INFO:journalctl@ceph.osd.2.smithi002.stdout:Sep 17 00:13:59 smithi002 sudo[38318]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-09-17T00:13:59.729 INFO:journalctl@ceph.osd.2.smithi002.stdout:Sep 17 00:13:59 smithi002 sudo[38318]: pam_unix(sudo:session): session closed for user root 2024-09-17T00:14:00.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:59 smithi002 sudo[38354]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-17T00:14:00.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:59 smithi002 sudo[38354]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-09-17T00:14:00.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:13:59 smithi002 sudo[38354]: pam_unix(sudo:session): session closed for user root 2024-09-17T00:14:00.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:00 smithi099 ceph-mon[24594]: pgmap v79: 1 pgs: 1 creating+peering; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-09-17T00:14:00.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:00 smithi099 ceph-mon[24594]: mgrmap e15: y(active, since 2m), standbys: x 2024-09-17T00:14:00.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:00 smithi002 ceph-mon[28203]: pgmap v79: 1 pgs: 1 creating+peering; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-09-17T00:14:00.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:00 smithi002 ceph-mon[28203]: mgrmap e15: y(active, since 2m), standbys: x 2024-09-17T00:14:00.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:00 smithi002 sudo[38358]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-17T00:14:00.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:00 smithi002 sudo[38358]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-09-17T00:14:00.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:00 smithi002 sudo[38358]: pam_unix(sudo:session): session closed for user root 2024-09-17T00:14:00.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:00 smithi002 ceph-mon[23737]: pgmap v79: 1 pgs: 1 creating+peering; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-09-17T00:14:00.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:00 smithi002 ceph-mon[23737]: mgrmap e15: y(active, since 2m), standbys: x 2024-09-17T00:14:01.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:00 smithi002 sudo[38395]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-17T00:14:01.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:00 smithi002 sudo[38395]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-09-17T00:14:01.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:00 smithi002 sudo[38395]: pam_unix(sudo:session): session closed for user root 2024-09-17T00:14:01.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:01 smithi002 sudo[38446]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-17T00:14:01.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:01 smithi002 sudo[38446]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-09-17T00:14:01.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:01 smithi002 sudo[38446]: pam_unix(sudo:session): session closed for user root 2024-09-17T00:14:01.770 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:01 smithi099 sudo[26094]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-17T00:14:01.770 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:01 smithi099 sudo[26094]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-09-17T00:14:01.771 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:01 smithi099 sudo[26094]: pam_unix(sudo:session): session closed for user root 2024-09-17T00:14:02.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:01 smithi099 sudo[26097]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-17T00:14:02.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:01 smithi099 sudo[26097]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-09-17T00:14:02.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:01 smithi099 sudo[26097]: pam_unix(sudo:session): session closed for user root 2024-09-17T00:14:02.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:01 smithi099 ceph-mon[24594]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-17T00:14:02.218 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:01 smithi099 ceph-mon[24594]: from='client.? 172.21.15.2:0/3859061362' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d9475026-f3ce-4b45-b5de-780c78dbc233"}]: dispatch 2024-09-17T00:14:02.218 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:01 smithi099 ceph-mon[24594]: pgmap v80: 1 pgs: 1 creating+peering; 0 B data, 16 MiB used, 268 GiB / 268 GiB avail 2024-09-17T00:14:02.218 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:01 smithi099 ceph-mon[24594]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-17T00:14:02.218 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:01 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T00:14:02.218 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:01 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T00:14:02.218 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:01 smithi099 ceph-mon[24594]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d9475026-f3ce-4b45-b5de-780c78dbc233"}]: dispatch 2024-09-17T00:14:02.218 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:01 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T00:14:02.218 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:01 smithi099 ceph-mon[24594]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d9475026-f3ce-4b45-b5de-780c78dbc233"}]': finished 2024-09-17T00:14:02.218 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:01 smithi099 ceph-mon[24594]: osdmap e23: 4 total, 3 up, 4 in 2024-09-17T00:14:02.218 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:01 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T00:14:02.218 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:01 smithi099 ceph-mon[24594]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-17T00:14:02.218 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:01 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T00:14:02.218 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:01 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T00:14:02.219 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:01 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T00:14:02.219 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:01 smithi099 ceph-mon[24594]: from='client.? 172.21.15.2:0/884123065' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-17T00:14:02.219 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:01 smithi099 ceph-mon[24594]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-17T00:14:02.219 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:01 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T00:14:02.219 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:01 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T00:14:02.219 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:01 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T00:14:02.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:01 smithi002 ceph-mon[23737]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-17T00:14:02.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:01 smithi002 ceph-mon[23737]: from='client.? 172.21.15.2:0/3859061362' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d9475026-f3ce-4b45-b5de-780c78dbc233"}]: dispatch 2024-09-17T00:14:02.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:01 smithi002 ceph-mon[23737]: pgmap v80: 1 pgs: 1 creating+peering; 0 B data, 16 MiB used, 268 GiB / 268 GiB avail 2024-09-17T00:14:02.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:01 smithi002 ceph-mon[23737]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-17T00:14:02.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:01 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T00:14:02.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:01 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T00:14:02.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:01 smithi002 ceph-mon[23737]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d9475026-f3ce-4b45-b5de-780c78dbc233"}]: dispatch 2024-09-17T00:14:02.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:01 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T00:14:02.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:01 smithi002 ceph-mon[23737]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d9475026-f3ce-4b45-b5de-780c78dbc233"}]': finished 2024-09-17T00:14:02.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:01 smithi002 ceph-mon[23737]: osdmap e23: 4 total, 3 up, 4 in 2024-09-17T00:14:02.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:01 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T00:14:02.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:01 smithi002 ceph-mon[23737]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-17T00:14:02.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:01 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T00:14:02.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:01 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T00:14:02.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:01 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T00:14:02.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:01 smithi002 ceph-mon[23737]: from='client.? 172.21.15.2:0/884123065' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-17T00:14:02.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:01 smithi002 ceph-mon[23737]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-17T00:14:02.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:01 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T00:14:02.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:01 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T00:14:02.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:01 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T00:14:02.231 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:01 smithi002 ceph-mon[28203]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-17T00:14:02.231 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:01 smithi002 ceph-mon[28203]: from='client.? 172.21.15.2:0/3859061362' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d9475026-f3ce-4b45-b5de-780c78dbc233"}]: dispatch 2024-09-17T00:14:02.231 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:01 smithi002 ceph-mon[28203]: pgmap v80: 1 pgs: 1 creating+peering; 0 B data, 16 MiB used, 268 GiB / 268 GiB avail 2024-09-17T00:14:02.231 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:01 smithi002 ceph-mon[28203]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-17T00:14:02.231 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:01 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T00:14:02.231 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:01 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T00:14:02.231 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:01 smithi002 ceph-mon[28203]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d9475026-f3ce-4b45-b5de-780c78dbc233"}]: dispatch 2024-09-17T00:14:02.231 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:01 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T00:14:02.231 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:01 smithi002 ceph-mon[28203]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d9475026-f3ce-4b45-b5de-780c78dbc233"}]': finished 2024-09-17T00:14:02.231 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:01 smithi002 ceph-mon[28203]: osdmap e23: 4 total, 3 up, 4 in 2024-09-17T00:14:02.231 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:01 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T00:14:02.231 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:01 smithi002 ceph-mon[28203]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-17T00:14:02.232 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:01 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T00:14:02.232 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:01 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T00:14:02.232 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:01 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T00:14:02.232 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:01 smithi002 ceph-mon[28203]: from='client.? 172.21.15.2:0/884123065' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-17T00:14:02.232 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:01 smithi002 ceph-mon[28203]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-17T00:14:02.232 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:01 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T00:14:02.232 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:01 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T00:14:02.232 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:01 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T00:14:03.162 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:02 smithi002 ceph-mon[23737]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-17T00:14:03.163 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:02 smithi002 ceph-mon[23737]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-17T00:14:03.163 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:02 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:14:03.163 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:02 smithi002 ceph-mon[28203]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-17T00:14:03.164 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:02 smithi002 ceph-mon[28203]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-17T00:14:03.164 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:02 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:14:03.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:02 smithi099 ceph-mon[24594]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-17T00:14:03.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:02 smithi099 ceph-mon[24594]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-17T00:14:03.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:02 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:14:04.137 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:03 smithi002 ceph-mon[28203]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-09-17T00:14:04.138 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:03 smithi002 ceph-mon[23737]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-09-17T00:14:04.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:03 smithi099 ceph-mon[24594]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-09-17T00:14:06.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:05 smithi099 ceph-mon[24594]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-09-17T00:14:06.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:05 smithi002 ceph-mon[23737]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-09-17T00:14:06.230 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:05 smithi002 ceph-mon[28203]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-09-17T00:14:06.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:06 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-09-17T00:14:06.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:06 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:14:06.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:06 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-09-17T00:14:06.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:06 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:14:07.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:06 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-09-17T00:14:07.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:06 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:14:08.095 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:07 smithi002 ceph-mon[28203]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 4.5 KiB/s rd, 196 KiB/s wr, 13 op/s 2024-09-17T00:14:08.095 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:07 smithi002 ceph-mon[28203]: Deploying daemon osd.3 on smithi002 2024-09-17T00:14:08.095 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:07 smithi002 ceph-mon[23737]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 4.5 KiB/s rd, 196 KiB/s wr, 13 op/s 2024-09-17T00:14:08.095 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:07 smithi002 ceph-mon[23737]: Deploying daemon osd.3 on smithi002 2024-09-17T00:14:08.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:07 smithi099 ceph-mon[24594]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 4.5 KiB/s rd, 196 KiB/s wr, 13 op/s 2024-09-17T00:14:08.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:07 smithi099 ceph-mon[24594]: Deploying daemon osd.3 on smithi002 2024-09-17T00:14:08.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:08 smithi002 ceph-mon[23737]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-09-17T00:14:08.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:08 smithi002 ceph-mon[28203]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-09-17T00:14:09.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:08 smithi099 ceph-mon[24594]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-09-17T00:14:11.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:11 smithi002 ceph-mon[28203]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-09-17T00:14:11.480 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:11 smithi002 ceph-mon[23737]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-09-17T00:14:11.716 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:11 smithi099 ceph-mon[24594]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-09-17T00:14:13.030 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:12 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:14:13.030 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:12 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T00:14:13.030 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:12 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:14:13.030 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:12 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:14:13.030 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:12 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:14:13.030 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:12 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T00:14:13.030 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:12 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:14:13.030 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:12 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:14:13.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:12 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:14:13.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:12 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T00:14:13.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:12 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:14:13.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:12 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:14:14.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:13 smithi099 ceph-mon[24594]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-09-17T00:14:14.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:13 smithi002 ceph-mon[28203]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-09-17T00:14:14.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:13 smithi002 ceph-mon[23737]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-09-17T00:14:15.045 INFO:teuthology.orchestra.run.smithi002.stdout:Created osd(s) 3 on host 'smithi002' 2024-09-17T00:14:15.676 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:15 smithi002 ceph-mon[28203]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-09-17T00:14:15.676 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:15 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:14:15.677 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:15 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:14:15.677 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:15 smithi002 ceph-mon[28203]: from='osd.3 [v2:172.21.15.2:6826/1628413766,v1:172.21.15.2:6827/1628413766]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-17T00:14:15.677 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:15 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:14:15.677 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:15 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T00:14:15.677 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:15 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:14:15.677 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:15 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:14:15.677 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:15 smithi002 ceph-mon[23737]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-09-17T00:14:15.677 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:15 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:14:15.677 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:15 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:14:15.677 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:15 smithi002 ceph-mon[23737]: from='osd.3 [v2:172.21.15.2:6826/1628413766,v1:172.21.15.2:6827/1628413766]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-17T00:14:15.678 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:15 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:14:15.678 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:15 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T00:14:15.678 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:15 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:14:15.678 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:15 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:14:15.701 DEBUG:teuthology.orchestra.run.smithi002:osd.3> sudo journalctl -f -n 0 -u ceph-3aad27a2-7489-11ef-bceb-c7b262605968@osd.3.service 2024-09-17T00:14:15.704 INFO:tasks.cephadm:Deploying osd.4 on smithi099 with /dev/vg_nvme/lv_4... 2024-09-17T00:14:15.704 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3aad27a2-7489-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_4 2024-09-17T00:14:15.716 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:15 smithi099 ceph-mon[24594]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-09-17T00:14:15.716 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:15 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:14:15.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:15 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:14:15.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:15 smithi099 ceph-mon[24594]: from='osd.3 [v2:172.21.15.2:6826/1628413766,v1:172.21.15.2:6827/1628413766]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-17T00:14:15.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:15 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:14:15.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:15 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T00:14:15.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:15 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:14:15.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:15 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:14:16.810 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:16 smithi002 ceph-mon[23737]: from='osd.3 [v2:172.21.15.2:6826/1628413766,v1:172.21.15.2:6827/1628413766]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-17T00:14:16.810 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:16 smithi002 ceph-mon[23737]: osdmap e24: 4 total, 3 up, 4 in 2024-09-17T00:14:16.810 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:16 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T00:14:16.811 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:16 smithi002 ceph-mon[23737]: from='osd.3 [v2:172.21.15.2:6826/1628413766,v1:172.21.15.2:6827/1628413766]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]: dispatch 2024-09-17T00:14:16.811 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:16 smithi002 ceph-mon[28203]: from='osd.3 [v2:172.21.15.2:6826/1628413766,v1:172.21.15.2:6827/1628413766]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-17T00:14:16.811 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:16 smithi002 ceph-mon[28203]: osdmap e24: 4 total, 3 up, 4 in 2024-09-17T00:14:16.811 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:16 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T00:14:16.811 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:16 smithi002 ceph-mon[28203]: from='osd.3 [v2:172.21.15.2:6826/1628413766,v1:172.21.15.2:6827/1628413766]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]: dispatch 2024-09-17T00:14:16.811 INFO:journalctl@ceph.osd.3.smithi002.stdout:Sep 17 00:14:16 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-3[39814]: 2024-09-17T00:14:16.799+0000 7f68cf925700 -1 osd.3 0 waiting for initial osdmap 2024-09-17T00:14:16.812 INFO:journalctl@ceph.osd.3.smithi002.stdout:Sep 17 00:14:16 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-3[39814]: 2024-09-17T00:14:16.804+0000 7f68cb2be700 -1 osd.3 25 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T00:14:17.051 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:16 smithi099 ceph-mon[24594]: from='osd.3 [v2:172.21.15.2:6826/1628413766,v1:172.21.15.2:6827/1628413766]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-17T00:14:17.051 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:16 smithi099 ceph-mon[24594]: osdmap e24: 4 total, 3 up, 4 in 2024-09-17T00:14:17.051 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:16 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T00:14:17.051 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:16 smithi099 ceph-mon[24594]: from='osd.3 [v2:172.21.15.2:6826/1628413766,v1:172.21.15.2:6827/1628413766]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]: dispatch 2024-09-17T00:14:17.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:17 smithi099 ceph-mon[24594]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-09-17T00:14:17.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:17 smithi099 ceph-mon[24594]: from='osd.3 [v2:172.21.15.2:6826/1628413766,v1:172.21.15.2:6827/1628413766]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]': finished 2024-09-17T00:14:17.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:17 smithi099 ceph-mon[24594]: osdmap e25: 4 total, 3 up, 4 in 2024-09-17T00:14:17.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:17 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T00:14:17.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:17 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T00:14:18.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:17 smithi002 ceph-mon[23737]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-09-17T00:14:18.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:17 smithi002 ceph-mon[23737]: from='osd.3 [v2:172.21.15.2:6826/1628413766,v1:172.21.15.2:6827/1628413766]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]': finished 2024-09-17T00:14:18.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:17 smithi002 ceph-mon[23737]: osdmap e25: 4 total, 3 up, 4 in 2024-09-17T00:14:18.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:17 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T00:14:18.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:17 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T00:14:18.230 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:17 smithi002 ceph-mon[28203]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-09-17T00:14:18.230 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:17 smithi002 ceph-mon[28203]: from='osd.3 [v2:172.21.15.2:6826/1628413766,v1:172.21.15.2:6827/1628413766]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]': finished 2024-09-17T00:14:18.230 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:17 smithi002 ceph-mon[28203]: osdmap e25: 4 total, 3 up, 4 in 2024-09-17T00:14:18.230 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:17 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T00:14:18.230 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:17 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T00:14:18.248 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-09-17T00:14:18.266 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3aad27a2-7489-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi099:vg_nvme/lv_4 2024-09-17T00:14:18.846 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:18 smithi099 ceph-mon[24594]: purged_snaps scrub starts 2024-09-17T00:14:18.847 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:18 smithi099 ceph-mon[24594]: purged_snaps scrub ok 2024-09-17T00:14:18.847 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:18 smithi099 ceph-mon[24594]: osd.3 [v2:172.21.15.2:6826/1628413766,v1:172.21.15.2:6827/1628413766] boot 2024-09-17T00:14:18.847 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:18 smithi099 ceph-mon[24594]: osdmap e26: 4 total, 4 up, 4 in 2024-09-17T00:14:18.847 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:18 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T00:14:18.847 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:18 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:14:18.847 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:18 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-17T00:14:18.847 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:18 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-17T00:14:18.847 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:18 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-17T00:14:18.847 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:18 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-17T00:14:18.847 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:18 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:14:18.847 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:18 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:14:18.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:18 smithi002 ceph-mon[23737]: purged_snaps scrub starts 2024-09-17T00:14:18.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:18 smithi002 ceph-mon[23737]: purged_snaps scrub ok 2024-09-17T00:14:18.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:18 smithi002 ceph-mon[23737]: osd.3 [v2:172.21.15.2:6826/1628413766,v1:172.21.15.2:6827/1628413766] boot 2024-09-17T00:14:18.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:18 smithi002 ceph-mon[23737]: osdmap e26: 4 total, 4 up, 4 in 2024-09-17T00:14:18.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:18 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T00:14:18.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:18 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:14:18.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:18 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-17T00:14:18.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:18 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-17T00:14:18.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:18 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-17T00:14:18.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:18 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-17T00:14:18.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:18 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:14:18.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:18 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:14:18.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:18 smithi002 ceph-mon[28203]: purged_snaps scrub starts 2024-09-17T00:14:18.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:18 smithi002 ceph-mon[28203]: purged_snaps scrub ok 2024-09-17T00:14:18.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:18 smithi002 ceph-mon[28203]: osd.3 [v2:172.21.15.2:6826/1628413766,v1:172.21.15.2:6827/1628413766] boot 2024-09-17T00:14:18.981 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:18 smithi002 ceph-mon[28203]: osdmap e26: 4 total, 4 up, 4 in 2024-09-17T00:14:18.981 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:18 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T00:14:18.981 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:18 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:14:18.981 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:18 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-17T00:14:18.981 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:18 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-17T00:14:18.981 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:18 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-17T00:14:18.981 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:18 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-17T00:14:18.981 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:18 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:14:18.981 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:18 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:14:20.105 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:19 smithi099 ceph-mon[24594]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-09-17T00:14:20.105 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:19 smithi099 ceph-mon[24594]: Detected new or changed devices on smithi002 2024-09-17T00:14:20.105 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:19 smithi099 ceph-mon[24594]: Adjusting osd_memory_target on smithi002 to 4017M 2024-09-17T00:14:20.105 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:19 smithi099 ceph-mon[24594]: osdmap e27: 4 total, 4 up, 4 in 2024-09-17T00:14:20.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:19 smithi002 ceph-mon[28203]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-09-17T00:14:20.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:19 smithi002 ceph-mon[28203]: Detected new or changed devices on smithi002 2024-09-17T00:14:20.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:19 smithi002 ceph-mon[28203]: Adjusting osd_memory_target on smithi002 to 4017M 2024-09-17T00:14:20.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:19 smithi002 ceph-mon[28203]: osdmap e27: 4 total, 4 up, 4 in 2024-09-17T00:14:20.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:19 smithi002 ceph-mon[23737]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-09-17T00:14:20.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:19 smithi002 ceph-mon[23737]: Detected new or changed devices on smithi002 2024-09-17T00:14:20.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:19 smithi002 ceph-mon[23737]: Adjusting osd_memory_target on smithi002 to 4017M 2024-09-17T00:14:20.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:19 smithi002 ceph-mon[23737]: osdmap e27: 4 total, 4 up, 4 in 2024-09-17T00:14:21.105 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:20 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-17T00:14:21.105 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:20 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-17T00:14:21.106 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:20 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:14:21.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:20 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-17T00:14:21.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:20 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-17T00:14:21.230 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:20 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:14:21.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:20 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-17T00:14:21.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:20 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-17T00:14:21.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:20 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:14:21.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:21 smithi099 ceph-mon[24594]: from='client.24218 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi099:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:14:21.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:21 smithi099 ceph-mon[24594]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-09-17T00:14:22.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:21 smithi002 ceph-mon[23737]: from='client.24218 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi099:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:14:22.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:21 smithi002 ceph-mon[23737]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-09-17T00:14:22.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:21 smithi002 ceph-mon[28203]: from='client.24218 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi099:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:14:22.230 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:21 smithi002 ceph-mon[28203]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-09-17T00:14:23.216 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:22 smithi099 ceph-mon[24594]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-09-17T00:14:23.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:22 smithi099 ceph-mon[24594]: from='client.? 172.21.15.99:0/26738382' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f8f0575a-3699-4e8f-b969-e4b6973708df"}]: dispatch 2024-09-17T00:14:23.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:22 smithi099 ceph-mon[24594]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f8f0575a-3699-4e8f-b969-e4b6973708df"}]: dispatch 2024-09-17T00:14:23.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:22 smithi099 ceph-mon[24594]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f8f0575a-3699-4e8f-b969-e4b6973708df"}]': finished 2024-09-17T00:14:23.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:22 smithi099 ceph-mon[24594]: osdmap e28: 5 total, 4 up, 5 in 2024-09-17T00:14:23.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:22 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T00:14:23.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:22 smithi002 ceph-mon[28203]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-09-17T00:14:23.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:22 smithi002 ceph-mon[28203]: from='client.? 172.21.15.99:0/26738382' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f8f0575a-3699-4e8f-b969-e4b6973708df"}]: dispatch 2024-09-17T00:14:23.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:22 smithi002 ceph-mon[28203]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f8f0575a-3699-4e8f-b969-e4b6973708df"}]: dispatch 2024-09-17T00:14:23.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:22 smithi002 ceph-mon[28203]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f8f0575a-3699-4e8f-b969-e4b6973708df"}]': finished 2024-09-17T00:14:23.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:22 smithi002 ceph-mon[28203]: osdmap e28: 5 total, 4 up, 5 in 2024-09-17T00:14:23.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:22 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T00:14:23.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:22 smithi002 ceph-mon[23737]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-09-17T00:14:23.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:22 smithi002 ceph-mon[23737]: from='client.? 172.21.15.99:0/26738382' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f8f0575a-3699-4e8f-b969-e4b6973708df"}]: dispatch 2024-09-17T00:14:23.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:22 smithi002 ceph-mon[23737]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f8f0575a-3699-4e8f-b969-e4b6973708df"}]: dispatch 2024-09-17T00:14:23.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:22 smithi002 ceph-mon[23737]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f8f0575a-3699-4e8f-b969-e4b6973708df"}]': finished 2024-09-17T00:14:23.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:22 smithi002 ceph-mon[23737]: osdmap e28: 5 total, 4 up, 5 in 2024-09-17T00:14:23.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:22 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T00:14:24.118 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:23 smithi099 ceph-mon[24594]: from='client.? 172.21.15.99:0/3138648077' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-17T00:14:24.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:23 smithi002 ceph-mon[28203]: from='client.? 172.21.15.99:0/3138648077' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-17T00:14:24.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:23 smithi002 ceph-mon[23737]: from='client.? 172.21.15.99:0/3138648077' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-17T00:14:25.100 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:24 smithi099 ceph-mon[24594]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-09-17T00:14:25.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:24 smithi002 ceph-mon[23737]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-09-17T00:14:25.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:24 smithi002 ceph-mon[28203]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-09-17T00:14:27.466 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:27 smithi099 ceph-mon[24594]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-09-17T00:14:27.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:27 smithi002 ceph-mon[23737]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-09-17T00:14:27.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:27 smithi002 ceph-mon[28203]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-09-17T00:14:28.863 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:28 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-09-17T00:14:28.864 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:28 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:14:28.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:28 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-09-17T00:14:28.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:28 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:14:28.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:28 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-09-17T00:14:28.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:28 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:14:29.794 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:29 smithi099 ceph-mon[24594]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-09-17T00:14:29.794 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:29 smithi099 ceph-mon[24594]: Deploying daemon osd.4 on smithi099 2024-09-17T00:14:29.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:29 smithi002 ceph-mon[23737]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-09-17T00:14:29.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:29 smithi002 ceph-mon[23737]: Deploying daemon osd.4 on smithi099 2024-09-17T00:14:29.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:29 smithi002 ceph-mon[28203]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-09-17T00:14:29.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:29 smithi002 ceph-mon[28203]: Deploying daemon osd.4 on smithi099 2024-09-17T00:14:31.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:31 smithi099 ceph-mon[24594]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-09-17T00:14:31.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:31 smithi002 ceph-mon[23737]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-09-17T00:14:31.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:31 smithi002 ceph-mon[28203]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-09-17T00:14:33.793 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:33 smithi099 ceph-mon[24594]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-09-17T00:14:33.793 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:33 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:14:33.793 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:33 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T00:14:33.793 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:33 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:14:33.793 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:33 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:14:33.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:33 smithi002 ceph-mon[23737]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-09-17T00:14:33.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:33 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:14:33.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:33 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T00:14:33.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:33 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:14:33.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:33 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:14:33.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:33 smithi002 ceph-mon[28203]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-09-17T00:14:33.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:33 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:14:33.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:33 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T00:14:33.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:33 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:14:33.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:33 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:14:35.608 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:35 smithi099 ceph-mon[24594]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-09-17T00:14:35.794 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:35 smithi002 ceph-mon[28203]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-09-17T00:14:35.795 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:35 smithi002 ceph-mon[23737]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-09-17T00:14:36.358 INFO:teuthology.orchestra.run.smithi099.stdout:Created osd(s) 4 on host 'smithi099' 2024-09-17T00:14:36.912 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:36 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:14:36.913 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:36 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:14:36.913 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:36 smithi099 ceph-mon[24594]: from='osd.4 [v2:172.21.15.99:6800/2601657006,v1:172.21.15.99:6801/2601657006]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-17T00:14:36.913 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:36 smithi099 ceph-mon[24594]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-17T00:14:36.934 DEBUG:teuthology.orchestra.run.smithi099:osd.4> sudo journalctl -f -n 0 -u ceph-3aad27a2-7489-11ef-bceb-c7b262605968@osd.4.service 2024-09-17T00:14:36.937 INFO:tasks.cephadm:Deploying osd.5 on smithi099 with /dev/vg_nvme/lv_3... 2024-09-17T00:14:36.937 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3aad27a2-7489-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_3 2024-09-17T00:14:36.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:36 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:14:36.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:36 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:14:36.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:36 smithi002 ceph-mon[23737]: from='osd.4 [v2:172.21.15.99:6800/2601657006,v1:172.21.15.99:6801/2601657006]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-17T00:14:36.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:36 smithi002 ceph-mon[23737]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-17T00:14:36.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:36 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:14:36.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:36 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:14:36.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:36 smithi002 ceph-mon[28203]: from='osd.4 [v2:172.21.15.99:6800/2601657006,v1:172.21.15.99:6801/2601657006]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-17T00:14:36.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:36 smithi002 ceph-mon[28203]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-17T00:14:37.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:37 smithi099 ceph-mon[24594]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-09-17T00:14:37.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:37 smithi099 ceph-mon[24594]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-17T00:14:37.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:37 smithi099 ceph-mon[24594]: osdmap e29: 5 total, 4 up, 5 in 2024-09-17T00:14:37.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:37 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T00:14:37.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:37 smithi099 ceph-mon[24594]: from='osd.4 [v2:172.21.15.99:6800/2601657006,v1:172.21.15.99:6801/2601657006]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi099", "root=default"]}]: dispatch 2024-09-17T00:14:37.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:37 smithi099 ceph-mon[24594]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi099", "root=default"]}]: dispatch 2024-09-17T00:14:37.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:37 smithi002 ceph-mon[23737]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-09-17T00:14:37.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:37 smithi002 ceph-mon[23737]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-17T00:14:37.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:37 smithi002 ceph-mon[23737]: osdmap e29: 5 total, 4 up, 5 in 2024-09-17T00:14:37.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:37 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T00:14:37.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:37 smithi002 ceph-mon[23737]: from='osd.4 [v2:172.21.15.99:6800/2601657006,v1:172.21.15.99:6801/2601657006]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi099", "root=default"]}]: dispatch 2024-09-17T00:14:37.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:37 smithi002 ceph-mon[23737]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi099", "root=default"]}]: dispatch 2024-09-17T00:14:37.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:37 smithi002 ceph-mon[28203]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-09-17T00:14:37.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:37 smithi002 ceph-mon[28203]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-17T00:14:37.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:37 smithi002 ceph-mon[28203]: osdmap e29: 5 total, 4 up, 5 in 2024-09-17T00:14:37.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:37 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T00:14:37.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:37 smithi002 ceph-mon[28203]: from='osd.4 [v2:172.21.15.99:6800/2601657006,v1:172.21.15.99:6801/2601657006]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi099", "root=default"]}]: dispatch 2024-09-17T00:14:37.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:37 smithi002 ceph-mon[28203]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi099", "root=default"]}]: dispatch 2024-09-17T00:14:38.698 INFO:journalctl@ceph.osd.4.smithi099.stdout:Sep 17 00:14:38 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-4[27932]: 2024-09-17T00:14:38.367+0000 7f36d10e8700 -1 osd.4 0 waiting for initial osdmap 2024-09-17T00:14:38.698 INFO:journalctl@ceph.osd.4.smithi099.stdout:Sep 17 00:14:38 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-4[27932]: 2024-09-17T00:14:38.371+0000 7f36cb27e700 -1 osd.4 30 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T00:14:39.701 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:39 smithi099 ceph-mon[24594]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-09-17T00:14:39.701 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:39 smithi099 ceph-mon[24594]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi099", "root=default"]}]': finished 2024-09-17T00:14:39.702 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:39 smithi099 ceph-mon[24594]: osdmap e30: 5 total, 4 up, 5 in 2024-09-17T00:14:39.702 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:39 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T00:14:39.702 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:39 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T00:14:39.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:39 smithi002 ceph-mon[23737]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-09-17T00:14:39.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:39 smithi002 ceph-mon[23737]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi099", "root=default"]}]': finished 2024-09-17T00:14:39.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:39 smithi002 ceph-mon[23737]: osdmap e30: 5 total, 4 up, 5 in 2024-09-17T00:14:39.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:39 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T00:14:39.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:39 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T00:14:39.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:39 smithi002 ceph-mon[28203]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-09-17T00:14:39.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:39 smithi002 ceph-mon[28203]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi099", "root=default"]}]': finished 2024-09-17T00:14:39.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:39 smithi002 ceph-mon[28203]: osdmap e30: 5 total, 4 up, 5 in 2024-09-17T00:14:39.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:39 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T00:14:39.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:39 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T00:14:40.554 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:40 smithi099 ceph-mon[24594]: purged_snaps scrub starts 2024-09-17T00:14:40.554 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:40 smithi099 ceph-mon[24594]: purged_snaps scrub ok 2024-09-17T00:14:40.554 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:40 smithi099 ceph-mon[24594]: osd.4 [v2:172.21.15.99:6800/2601657006,v1:172.21.15.99:6801/2601657006] boot 2024-09-17T00:14:40.555 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:40 smithi099 ceph-mon[24594]: osdmap e31: 5 total, 5 up, 5 in 2024-09-17T00:14:40.555 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:40 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T00:14:40.555 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:40 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:14:40.555 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:40 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-17T00:14:40.555 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:40 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:14:40.555 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:40 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:14:40.555 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:40 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T00:14:40.555 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:40 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:14:40.555 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:40 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:14:40.555 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:40 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:14:40.555 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:40 smithi099 ceph-mon[24594]: osdmap e32: 5 total, 5 up, 5 in 2024-09-17T00:14:40.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:40 smithi002 ceph-mon[23737]: purged_snaps scrub starts 2024-09-17T00:14:40.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:40 smithi002 ceph-mon[23737]: purged_snaps scrub ok 2024-09-17T00:14:40.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:40 smithi002 ceph-mon[23737]: osd.4 [v2:172.21.15.99:6800/2601657006,v1:172.21.15.99:6801/2601657006] boot 2024-09-17T00:14:40.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:40 smithi002 ceph-mon[23737]: osdmap e31: 5 total, 5 up, 5 in 2024-09-17T00:14:40.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:40 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T00:14:40.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:40 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:14:40.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:40 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-17T00:14:40.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:40 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:14:40.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:40 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:14:40.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:40 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T00:14:40.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:40 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:14:40.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:40 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:14:40.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:40 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:14:40.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:40 smithi002 ceph-mon[23737]: osdmap e32: 5 total, 5 up, 5 in 2024-09-17T00:14:40.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:40 smithi002 ceph-mon[28203]: purged_snaps scrub starts 2024-09-17T00:14:40.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:40 smithi002 ceph-mon[28203]: purged_snaps scrub ok 2024-09-17T00:14:40.731 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:40 smithi002 ceph-mon[28203]: osd.4 [v2:172.21.15.99:6800/2601657006,v1:172.21.15.99:6801/2601657006] boot 2024-09-17T00:14:40.731 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:40 smithi002 ceph-mon[28203]: osdmap e31: 5 total, 5 up, 5 in 2024-09-17T00:14:40.731 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:40 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T00:14:40.731 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:40 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:14:40.731 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:40 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-17T00:14:40.731 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:40 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:14:40.731 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:40 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:14:40.731 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:40 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T00:14:40.731 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:40 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:14:40.731 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:40 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:14:40.731 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:40 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:14:40.731 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:40 smithi002 ceph-mon[28203]: osdmap e32: 5 total, 5 up, 5 in 2024-09-17T00:14:41.664 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:41 smithi099 ceph-mon[24594]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-09-17T00:14:41.664 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:41 smithi099 ceph-mon[24594]: Detected new or changed devices on smithi099 2024-09-17T00:14:41.664 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:41 smithi099 ceph-mon[24594]: Adjusting osd_memory_target on smithi099 to 17088M 2024-09-17T00:14:41.664 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:41 smithi099 ceph-mon[24594]: osdmap e33: 5 total, 5 up, 5 in 2024-09-17T00:14:41.669 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-09-17T00:14:41.687 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3aad27a2-7489-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi099:vg_nvme/lv_3 2024-09-17T00:14:41.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:41 smithi002 ceph-mon[23737]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-09-17T00:14:41.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:41 smithi002 ceph-mon[23737]: Detected new or changed devices on smithi099 2024-09-17T00:14:41.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:41 smithi002 ceph-mon[23737]: Adjusting osd_memory_target on smithi099 to 17088M 2024-09-17T00:14:41.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:41 smithi002 ceph-mon[23737]: osdmap e33: 5 total, 5 up, 5 in 2024-09-17T00:14:41.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:41 smithi002 ceph-mon[28203]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-09-17T00:14:41.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:41 smithi002 ceph-mon[28203]: Detected new or changed devices on smithi099 2024-09-17T00:14:41.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:41 smithi002 ceph-mon[28203]: Adjusting osd_memory_target on smithi099 to 17088M 2024-09-17T00:14:41.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:41 smithi002 ceph-mon[28203]: osdmap e33: 5 total, 5 up, 5 in 2024-09-17T00:14:43.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:43 smithi099 ceph-mon[24594]: pgmap v112: 1 pgs: 1 remapped+peering; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-09-17T00:14:43.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:43 smithi099 ceph-mon[24594]: osdmap e34: 5 total, 5 up, 5 in 2024-09-17T00:14:43.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:43 smithi002 ceph-mon[23737]: pgmap v112: 1 pgs: 1 remapped+peering; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-09-17T00:14:43.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:43 smithi002 ceph-mon[23737]: osdmap e34: 5 total, 5 up, 5 in 2024-09-17T00:14:43.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:43 smithi002 ceph-mon[28203]: pgmap v112: 1 pgs: 1 remapped+peering; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-09-17T00:14:43.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:43 smithi002 ceph-mon[28203]: osdmap e34: 5 total, 5 up, 5 in 2024-09-17T00:14:44.688 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:44 smithi099 ceph-mon[24594]: osdmap e35: 5 total, 5 up, 5 in 2024-09-17T00:14:44.689 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:44 smithi099 ceph-mon[24594]: from='client.24245 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi099:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:14:44.689 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:44 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-17T00:14:44.689 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:44 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-17T00:14:44.689 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:44 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:14:44.689 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:44 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T00:14:44.689 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:44 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T00:14:44.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:44 smithi002 ceph-mon[23737]: osdmap e35: 5 total, 5 up, 5 in 2024-09-17T00:14:44.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:44 smithi002 ceph-mon[23737]: from='client.24245 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi099:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:14:44.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:44 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-17T00:14:44.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:44 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-17T00:14:44.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:44 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:14:44.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:44 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T00:14:44.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:44 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T00:14:44.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:44 smithi002 ceph-mon[28203]: osdmap e35: 5 total, 5 up, 5 in 2024-09-17T00:14:44.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:44 smithi002 ceph-mon[28203]: from='client.24245 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi099:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:14:44.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:44 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-17T00:14:44.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:44 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-17T00:14:44.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:44 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:14:44.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:44 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T00:14:44.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:44 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T00:14:45.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:45 smithi099 ceph-mon[24594]: pgmap v115: 1 pgs: 1 remapped+peering; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-09-17T00:14:45.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:45 smithi002 ceph-mon[23737]: pgmap v115: 1 pgs: 1 remapped+peering; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-09-17T00:14:45.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:45 smithi002 ceph-mon[28203]: pgmap v115: 1 pgs: 1 remapped+peering; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-09-17T00:14:46.721 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:46 smithi099 ceph-mon[24594]: from='client.? 172.21.15.99:0/1757777048' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c0ba894b-bcbd-40e5-b00d-73dbddc87d59"}]: dispatch 2024-09-17T00:14:46.721 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:46 smithi099 ceph-mon[24594]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c0ba894b-bcbd-40e5-b00d-73dbddc87d59"}]: dispatch 2024-09-17T00:14:46.721 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:46 smithi099 ceph-mon[24594]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c0ba894b-bcbd-40e5-b00d-73dbddc87d59"}]': finished 2024-09-17T00:14:46.721 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:46 smithi099 ceph-mon[24594]: osdmap e36: 6 total, 5 up, 6 in 2024-09-17T00:14:46.721 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:46 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T00:14:46.721 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:46 smithi099 ceph-mon[24594]: from='client.? 172.21.15.99:0/3476801484' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-17T00:14:46.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:46 smithi002 ceph-mon[23737]: from='client.? 172.21.15.99:0/1757777048' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c0ba894b-bcbd-40e5-b00d-73dbddc87d59"}]: dispatch 2024-09-17T00:14:46.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:46 smithi002 ceph-mon[23737]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c0ba894b-bcbd-40e5-b00d-73dbddc87d59"}]: dispatch 2024-09-17T00:14:46.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:46 smithi002 ceph-mon[23737]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c0ba894b-bcbd-40e5-b00d-73dbddc87d59"}]': finished 2024-09-17T00:14:46.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:46 smithi002 ceph-mon[23737]: osdmap e36: 6 total, 5 up, 6 in 2024-09-17T00:14:46.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:46 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T00:14:46.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:46 smithi002 ceph-mon[23737]: from='client.? 172.21.15.99:0/3476801484' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-17T00:14:46.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:46 smithi002 ceph-mon[28203]: from='client.? 172.21.15.99:0/1757777048' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c0ba894b-bcbd-40e5-b00d-73dbddc87d59"}]: dispatch 2024-09-17T00:14:46.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:46 smithi002 ceph-mon[28203]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c0ba894b-bcbd-40e5-b00d-73dbddc87d59"}]: dispatch 2024-09-17T00:14:46.731 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:46 smithi002 ceph-mon[28203]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c0ba894b-bcbd-40e5-b00d-73dbddc87d59"}]': finished 2024-09-17T00:14:46.731 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:46 smithi002 ceph-mon[28203]: osdmap e36: 6 total, 5 up, 6 in 2024-09-17T00:14:46.731 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:46 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T00:14:46.731 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:46 smithi002 ceph-mon[28203]: from='client.? 172.21.15.99:0/3476801484' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-17T00:14:47.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:47 smithi099 ceph-mon[24594]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 107 KiB/s, 0 objects/s recovering 2024-09-17T00:14:47.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:47 smithi002 ceph-mon[23737]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 107 KiB/s, 0 objects/s recovering 2024-09-17T00:14:47.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:47 smithi002 ceph-mon[28203]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 107 KiB/s, 0 objects/s recovering 2024-09-17T00:14:49.691 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:49 smithi099 ceph-mon[24594]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-09-17T00:14:49.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:49 smithi002 ceph-mon[23737]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-09-17T00:14:49.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:49 smithi002 ceph-mon[28203]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-09-17T00:14:51.966 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:51 smithi099 ceph-mon[24594]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-09-17T00:14:51.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:51 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-09-17T00:14:51.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:51 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:14:51.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:51 smithi002 ceph-mon[23737]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-09-17T00:14:51.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:51 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-09-17T00:14:51.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:51 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:14:51.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:51 smithi002 ceph-mon[28203]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-09-17T00:14:51.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:51 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-09-17T00:14:51.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:51 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:14:52.875 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:52 smithi099 ceph-mon[24594]: Deploying daemon osd.5 on smithi099 2024-09-17T00:14:52.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:52 smithi002 ceph-mon[23737]: Deploying daemon osd.5 on smithi099 2024-09-17T00:14:52.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:52 smithi002 ceph-mon[28203]: Deploying daemon osd.5 on smithi099 2024-09-17T00:14:53.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:53 smithi099 ceph-mon[24594]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-09-17T00:14:53.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:53 smithi002 ceph-mon[23737]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-09-17T00:14:53.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:53 smithi002 ceph-mon[28203]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-09-17T00:14:55.960 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:55 smithi099 ceph-mon[24594]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-09-17T00:14:55.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:55 smithi002 ceph-mon[23737]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-09-17T00:14:55.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:55 smithi002 ceph-mon[28203]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-09-17T00:14:57.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:57 smithi099 ceph-mon[24594]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 49 KiB/s, 0 objects/s recovering 2024-09-17T00:14:57.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:57 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:14:57.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:57 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T00:14:57.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:57 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:14:57.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:57 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:14:57.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:57 smithi002 ceph-mon[28203]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 49 KiB/s, 0 objects/s recovering 2024-09-17T00:14:57.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:57 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:14:57.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:57 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T00:14:57.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:57 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:14:57.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:57 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:14:57.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:57 smithi002 ceph-mon[23737]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 49 KiB/s, 0 objects/s recovering 2024-09-17T00:14:57.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:57 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:14:57.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:57 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T00:14:57.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:57 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:14:57.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:57 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:14:59.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:59 smithi099 ceph-mon[24594]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-09-17T00:14:59.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:59 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:14:59.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:59 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:14:59.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:59 smithi099 ceph-mon[24594]: from='osd.5 [v2:172.21.15.99:6808/1930951601,v1:172.21.15.99:6809/1930951601]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-17T00:14:59.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:14:59 smithi099 ceph-mon[24594]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-17T00:14:59.751 INFO:teuthology.orchestra.run.smithi099.stdout:Created osd(s) 5 on host 'smithi099' 2024-09-17T00:14:59.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:59 smithi002 ceph-mon[23737]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-09-17T00:14:59.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:59 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:14:59.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:59 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:14:59.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:59 smithi002 ceph-mon[23737]: from='osd.5 [v2:172.21.15.99:6808/1930951601,v1:172.21.15.99:6809/1930951601]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-17T00:14:59.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:14:59 smithi002 ceph-mon[23737]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-17T00:14:59.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:59 smithi002 ceph-mon[28203]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-09-17T00:14:59.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:59 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:14:59.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:59 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:14:59.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:59 smithi002 ceph-mon[28203]: from='osd.5 [v2:172.21.15.99:6808/1930951601,v1:172.21.15.99:6809/1930951601]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-17T00:14:59.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:14:59 smithi002 ceph-mon[28203]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-17T00:15:00.354 DEBUG:teuthology.orchestra.run.smithi099:osd.5> sudo journalctl -f -n 0 -u ceph-3aad27a2-7489-11ef-bceb-c7b262605968@osd.5.service 2024-09-17T00:15:00.356 INFO:tasks.cephadm:Deploying osd.6 on smithi099 with /dev/vg_nvme/lv_2... 2024-09-17T00:15:00.357 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3aad27a2-7489-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_2 2024-09-17T00:15:00.951 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:00 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:15:00.951 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:00 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T00:15:00.951 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:00 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:15:00.951 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:00 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:15:00.951 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:00 smithi099 ceph-mon[24594]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-17T00:15:00.952 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:00 smithi099 ceph-mon[24594]: osdmap e37: 6 total, 5 up, 6 in 2024-09-17T00:15:00.952 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:00 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T00:15:00.952 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:00 smithi099 ceph-mon[24594]: from='osd.5 [v2:172.21.15.99:6808/1930951601,v1:172.21.15.99:6809/1930951601]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi099", "root=default"]}]: dispatch 2024-09-17T00:15:00.952 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:00 smithi099 ceph-mon[24594]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi099", "root=default"]}]: dispatch 2024-09-17T00:15:01.202 INFO:journalctl@ceph.osd.5.smithi099.stdout:Sep 17 00:15:00 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-5[30851]: 2024-09-17T00:15:00.953+0000 7f5cae8c4700 -1 osd.5 0 waiting for initial osdmap 2024-09-17T00:15:01.203 INFO:journalctl@ceph.osd.5.smithi099.stdout:Sep 17 00:15:00 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-5[30851]: 2024-09-17T00:15:00.958+0000 7f5ca9a5c700 -1 osd.5 38 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T00:15:01.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:00 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:15:01.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:00 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T00:15:01.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:00 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:15:01.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:00 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:15:01.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:00 smithi002 ceph-mon[28203]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-17T00:15:01.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:00 smithi002 ceph-mon[28203]: osdmap e37: 6 total, 5 up, 6 in 2024-09-17T00:15:01.230 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:00 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T00:15:01.230 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:00 smithi002 ceph-mon[28203]: from='osd.5 [v2:172.21.15.99:6808/1930951601,v1:172.21.15.99:6809/1930951601]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi099", "root=default"]}]: dispatch 2024-09-17T00:15:01.230 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:00 smithi002 ceph-mon[28203]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi099", "root=default"]}]: dispatch 2024-09-17T00:15:01.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:00 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:15:01.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:00 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T00:15:01.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:00 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:15:01.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:00 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:15:01.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:00 smithi002 ceph-mon[23737]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-17T00:15:01.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:00 smithi002 ceph-mon[23737]: osdmap e37: 6 total, 5 up, 6 in 2024-09-17T00:15:01.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:00 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T00:15:01.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:00 smithi002 ceph-mon[23737]: from='osd.5 [v2:172.21.15.99:6808/1930951601,v1:172.21.15.99:6809/1930951601]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi099", "root=default"]}]: dispatch 2024-09-17T00:15:01.231 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:00 smithi002 ceph-mon[23737]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi099", "root=default"]}]: dispatch 2024-09-17T00:15:02.037 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:01 smithi099 ceph-mon[24594]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-09-17T00:15:02.037 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:01 smithi099 ceph-mon[24594]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi099", "root=default"]}]': finished 2024-09-17T00:15:02.037 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:01 smithi099 ceph-mon[24594]: osdmap e38: 6 total, 5 up, 6 in 2024-09-17T00:15:02.037 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:01 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T00:15:02.037 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:01 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T00:15:02.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:01 smithi002 ceph-mon[28203]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-09-17T00:15:02.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:01 smithi002 ceph-mon[28203]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi099", "root=default"]}]': finished 2024-09-17T00:15:02.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:01 smithi002 ceph-mon[28203]: osdmap e38: 6 total, 5 up, 6 in 2024-09-17T00:15:02.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:01 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T00:15:02.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:01 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T00:15:02.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:01 smithi002 ceph-mon[23737]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-09-17T00:15:02.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:01 smithi002 ceph-mon[23737]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi099", "root=default"]}]': finished 2024-09-17T00:15:02.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:01 smithi002 ceph-mon[23737]: osdmap e38: 6 total, 5 up, 6 in 2024-09-17T00:15:02.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:01 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T00:15:02.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:01 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T00:15:03.216 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:02 smithi099 ceph-mon[24594]: purged_snaps scrub starts 2024-09-17T00:15:03.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:02 smithi099 ceph-mon[24594]: purged_snaps scrub ok 2024-09-17T00:15:03.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:02 smithi099 ceph-mon[24594]: osd.5 [v2:172.21.15.99:6808/1930951601,v1:172.21.15.99:6809/1930951601] boot 2024-09-17T00:15:03.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:02 smithi099 ceph-mon[24594]: osdmap e39: 6 total, 6 up, 6 in 2024-09-17T00:15:03.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:02 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T00:15:03.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:02 smithi099 ceph-mon[24594]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-09-17T00:15:03.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:02 smithi099 ceph-mon[24594]: osdmap e40: 6 total, 6 up, 6 in 2024-09-17T00:15:03.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:02 smithi002 ceph-mon[28203]: purged_snaps scrub starts 2024-09-17T00:15:03.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:02 smithi002 ceph-mon[28203]: purged_snaps scrub ok 2024-09-17T00:15:03.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:02 smithi002 ceph-mon[28203]: osd.5 [v2:172.21.15.99:6808/1930951601,v1:172.21.15.99:6809/1930951601] boot 2024-09-17T00:15:03.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:02 smithi002 ceph-mon[28203]: osdmap e39: 6 total, 6 up, 6 in 2024-09-17T00:15:03.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:02 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T00:15:03.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:02 smithi002 ceph-mon[28203]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-09-17T00:15:03.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:02 smithi002 ceph-mon[28203]: osdmap e40: 6 total, 6 up, 6 in 2024-09-17T00:15:03.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:02 smithi002 ceph-mon[23737]: purged_snaps scrub starts 2024-09-17T00:15:03.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:02 smithi002 ceph-mon[23737]: purged_snaps scrub ok 2024-09-17T00:15:03.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:02 smithi002 ceph-mon[23737]: osd.5 [v2:172.21.15.99:6808/1930951601,v1:172.21.15.99:6809/1930951601] boot 2024-09-17T00:15:03.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:02 smithi002 ceph-mon[23737]: osdmap e39: 6 total, 6 up, 6 in 2024-09-17T00:15:03.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:02 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T00:15:03.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:02 smithi002 ceph-mon[23737]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-09-17T00:15:03.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:02 smithi002 ceph-mon[23737]: osdmap e40: 6 total, 6 up, 6 in 2024-09-17T00:15:04.694 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:04 smithi099 ceph-mon[24594]: osdmap e41: 6 total, 6 up, 6 in 2024-09-17T00:15:04.695 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:04 smithi099 ceph-mon[24594]: Detected new or changed devices on smithi099 2024-09-17T00:15:04.695 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:04 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:15:04.695 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:04 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-17T00:15:04.695 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:04 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-17T00:15:04.695 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:04 smithi099 ceph-mon[24594]: Adjusting osd_memory_target on smithi099 to 8544M 2024-09-17T00:15:04.695 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:04 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:15:04.695 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:04 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:15:04.966 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-09-17T00:15:04.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:04 smithi002 ceph-mon[23737]: osdmap e41: 6 total, 6 up, 6 in 2024-09-17T00:15:04.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:04 smithi002 ceph-mon[23737]: Detected new or changed devices on smithi099 2024-09-17T00:15:04.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:04 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:15:04.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:04 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-17T00:15:04.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:04 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-17T00:15:04.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:04 smithi002 ceph-mon[23737]: Adjusting osd_memory_target on smithi099 to 8544M 2024-09-17T00:15:04.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:04 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:15:04.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:04 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:15:04.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:04 smithi002 ceph-mon[28203]: osdmap e41: 6 total, 6 up, 6 in 2024-09-17T00:15:04.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:04 smithi002 ceph-mon[28203]: Detected new or changed devices on smithi099 2024-09-17T00:15:04.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:04 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:15:04.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:04 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-17T00:15:04.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:04 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-17T00:15:04.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:04 smithi002 ceph-mon[28203]: Adjusting osd_memory_target on smithi099 to 8544M 2024-09-17T00:15:04.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:04 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:15:04.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:04 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:15:04.985 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3aad27a2-7489-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi099:vg_nvme/lv_2 2024-09-17T00:15:05.881 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:05 smithi002 ceph-mon[28203]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-09-17T00:15:05.882 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:05 smithi002 ceph-mon[28203]: osdmap e42: 6 total, 6 up, 6 in 2024-09-17T00:15:05.882 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:05 smithi002 ceph-mon[23737]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-09-17T00:15:05.882 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:05 smithi002 ceph-mon[23737]: osdmap e42: 6 total, 6 up, 6 in 2024-09-17T00:15:05.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:05 smithi099 ceph-mon[24594]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-09-17T00:15:05.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:05 smithi099 ceph-mon[24594]: osdmap e42: 6 total, 6 up, 6 in 2024-09-17T00:15:06.656 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:06 smithi099 ceph-mon[24594]: osdmap e43: 6 total, 6 up, 6 in 2024-09-17T00:15:06.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:06 smithi002 ceph-mon[23737]: osdmap e43: 6 total, 6 up, 6 in 2024-09-17T00:15:06.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:06 smithi002 ceph-mon[28203]: osdmap e43: 6 total, 6 up, 6 in 2024-09-17T00:15:07.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:07 smithi099 ceph-mon[24594]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-09-17T00:15:07.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:07 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-17T00:15:07.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:07 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-17T00:15:07.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:07 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:15:07.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:07 smithi002 ceph-mon[23737]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-09-17T00:15:07.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:07 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-17T00:15:07.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:07 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-17T00:15:07.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:07 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:15:07.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:07 smithi002 ceph-mon[28203]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-09-17T00:15:07.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:07 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-17T00:15:07.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:07 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-17T00:15:07.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:07 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:15:08.662 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:08 smithi099 ceph-mon[24594]: from='client.24272 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi099:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:15:08.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:08 smithi002 ceph-mon[23737]: from='client.24272 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi099:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:15:08.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:08 smithi002 ceph-mon[28203]: from='client.24272 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi099:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:15:09.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:09 smithi099 ceph-mon[24594]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-09-17T00:15:09.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:09 smithi099 ceph-mon[24594]: from='client.? 172.21.15.99:0/3084854784' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9aeb407a-4fd9-415c-aa66-1e66d8aaa700"}]: dispatch 2024-09-17T00:15:09.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:09 smithi099 ceph-mon[24594]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9aeb407a-4fd9-415c-aa66-1e66d8aaa700"}]: dispatch 2024-09-17T00:15:09.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:09 smithi099 ceph-mon[24594]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9aeb407a-4fd9-415c-aa66-1e66d8aaa700"}]': finished 2024-09-17T00:15:09.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:09 smithi099 ceph-mon[24594]: osdmap e44: 7 total, 6 up, 7 in 2024-09-17T00:15:09.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:09 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T00:15:09.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:09 smithi002 ceph-mon[23737]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-09-17T00:15:09.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:09 smithi002 ceph-mon[23737]: from='client.? 172.21.15.99:0/3084854784' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9aeb407a-4fd9-415c-aa66-1e66d8aaa700"}]: dispatch 2024-09-17T00:15:09.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:09 smithi002 ceph-mon[23737]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9aeb407a-4fd9-415c-aa66-1e66d8aaa700"}]: dispatch 2024-09-17T00:15:09.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:09 smithi002 ceph-mon[23737]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9aeb407a-4fd9-415c-aa66-1e66d8aaa700"}]': finished 2024-09-17T00:15:09.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:09 smithi002 ceph-mon[23737]: osdmap e44: 7 total, 6 up, 7 in 2024-09-17T00:15:09.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:09 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T00:15:09.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:09 smithi002 ceph-mon[28203]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-09-17T00:15:09.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:09 smithi002 ceph-mon[28203]: from='client.? 172.21.15.99:0/3084854784' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9aeb407a-4fd9-415c-aa66-1e66d8aaa700"}]: dispatch 2024-09-17T00:15:09.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:09 smithi002 ceph-mon[28203]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9aeb407a-4fd9-415c-aa66-1e66d8aaa700"}]: dispatch 2024-09-17T00:15:09.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:09 smithi002 ceph-mon[28203]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9aeb407a-4fd9-415c-aa66-1e66d8aaa700"}]': finished 2024-09-17T00:15:09.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:09 smithi002 ceph-mon[28203]: osdmap e44: 7 total, 6 up, 7 in 2024-09-17T00:15:09.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:09 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T00:15:10.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:10 smithi099 ceph-mon[24594]: from='client.? 172.21.15.99:0/4041656193' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-17T00:15:10.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:10 smithi002 ceph-mon[23737]: from='client.? 172.21.15.99:0/4041656193' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-17T00:15:10.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:10 smithi002 ceph-mon[28203]: from='client.? 172.21.15.99:0/4041656193' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-17T00:15:11.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:11 smithi099 ceph-mon[24594]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-09-17T00:15:11.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:11 smithi002 ceph-mon[23737]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-09-17T00:15:11.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:11 smithi002 ceph-mon[28203]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-09-17T00:15:13.624 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:13 smithi099 ceph-mon[24594]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-09-17T00:15:13.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:13 smithi002 ceph-mon[28203]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-09-17T00:15:13.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:13 smithi002 ceph-mon[23737]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-09-17T00:15:15.966 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:15 smithi099 ceph-mon[24594]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-09-17T00:15:15.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:15 smithi002 ceph-mon[23737]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-09-17T00:15:15.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:15 smithi002 ceph-mon[28203]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-09-17T00:15:16.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:16 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-09-17T00:15:16.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:16 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:15:16.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:16 smithi099 ceph-mon[24594]: Deploying daemon osd.6 on smithi099 2024-09-17T00:15:16.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:16 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-09-17T00:15:16.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:16 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:15:16.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:16 smithi002 ceph-mon[23737]: Deploying daemon osd.6 on smithi099 2024-09-17T00:15:16.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:16 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-09-17T00:15:16.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:16 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:15:16.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:16 smithi002 ceph-mon[28203]: Deploying daemon osd.6 on smithi099 2024-09-17T00:15:17.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:17 smithi099 ceph-mon[24594]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-09-17T00:15:17.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:17 smithi002 ceph-mon[23737]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-09-17T00:15:17.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:17 smithi002 ceph-mon[28203]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-09-17T00:15:19.720 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:19 smithi099 ceph-mon[24594]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-09-17T00:15:19.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:19 smithi002 ceph-mon[23737]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-09-17T00:15:19.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:19 smithi002 ceph-mon[28203]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-09-17T00:15:21.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:21 smithi099 ceph-mon[24594]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-09-17T00:15:21.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:21 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:15:21.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:21 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T00:15:21.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:21 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:15:21.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:21 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:15:21.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:21 smithi002 ceph-mon[23737]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-09-17T00:15:21.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:21 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:15:21.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:21 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T00:15:21.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:21 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:15:21.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:21 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:15:21.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:21 smithi002 ceph-mon[28203]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-09-17T00:15:21.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:21 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:15:21.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:21 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T00:15:21.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:21 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:15:21.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:21 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:15:23.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:23 smithi099 ceph-mon[24594]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-09-17T00:15:23.777 INFO:teuthology.orchestra.run.smithi099.stdout:Created osd(s) 6 on host 'smithi099' 2024-09-17T00:15:23.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:23 smithi002 ceph-mon[28203]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-09-17T00:15:23.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:23 smithi002 ceph-mon[23737]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-09-17T00:15:24.438 DEBUG:teuthology.orchestra.run.smithi099:osd.6> sudo journalctl -f -n 0 -u ceph-3aad27a2-7489-11ef-bceb-c7b262605968@osd.6.service 2024-09-17T00:15:24.440 INFO:tasks.cephadm:Deploying osd.7 on smithi099 with /dev/vg_nvme/lv_1... 2024-09-17T00:15:24.441 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3aad27a2-7489-11ef-bceb-c7b262605968 -- lvm zap /dev/vg_nvme/lv_1 2024-09-17T00:15:24.716 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:24 smithi099 ceph-mon[24594]: from='osd.6 [v2:172.21.15.99:6816/462153986,v1:172.21.15.99:6817/462153986]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-17T00:15:24.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:24 smithi099 ceph-mon[24594]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-17T00:15:24.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:24 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:15:24.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:24 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:15:24.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:24 smithi002 ceph-mon[23737]: from='osd.6 [v2:172.21.15.99:6816/462153986,v1:172.21.15.99:6817/462153986]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-17T00:15:24.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:24 smithi002 ceph-mon[23737]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-17T00:15:24.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:24 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:15:24.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:24 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:15:24.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:24 smithi002 ceph-mon[28203]: from='osd.6 [v2:172.21.15.99:6816/462153986,v1:172.21.15.99:6817/462153986]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-17T00:15:24.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:24 smithi002 ceph-mon[28203]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-17T00:15:24.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:24 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:15:24.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:24 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:15:25.740 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:25 smithi099 ceph-mon[24594]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-09-17T00:15:25.740 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:25 smithi099 ceph-mon[24594]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-17T00:15:25.740 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:25 smithi099 ceph-mon[24594]: osdmap e45: 7 total, 6 up, 7 in 2024-09-17T00:15:25.740 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:25 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T00:15:25.740 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:25 smithi099 ceph-mon[24594]: from='osd.6 [v2:172.21.15.99:6816/462153986,v1:172.21.15.99:6817/462153986]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi099", "root=default"]}]: dispatch 2024-09-17T00:15:25.740 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:25 smithi099 ceph-mon[24594]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi099", "root=default"]}]: dispatch 2024-09-17T00:15:25.740 INFO:journalctl@ceph.osd.6.smithi099.stdout:Sep 17 00:15:25 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-6[33723]: 2024-09-17T00:15:25.643+0000 7ff82eadc700 -1 osd.6 0 waiting for initial osdmap 2024-09-17T00:15:25.741 INFO:journalctl@ceph.osd.6.smithi099.stdout:Sep 17 00:15:25 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-6[33723]: 2024-09-17T00:15:25.649+0000 7ff82746f700 -1 osd.6 46 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T00:15:25.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:25 smithi002 ceph-mon[23737]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-09-17T00:15:25.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:25 smithi002 ceph-mon[23737]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-17T00:15:25.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:25 smithi002 ceph-mon[23737]: osdmap e45: 7 total, 6 up, 7 in 2024-09-17T00:15:25.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:25 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T00:15:25.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:25 smithi002 ceph-mon[23737]: from='osd.6 [v2:172.21.15.99:6816/462153986,v1:172.21.15.99:6817/462153986]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi099", "root=default"]}]: dispatch 2024-09-17T00:15:25.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:25 smithi002 ceph-mon[23737]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi099", "root=default"]}]: dispatch 2024-09-17T00:15:25.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:25 smithi002 ceph-mon[28203]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-09-17T00:15:25.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:25 smithi002 ceph-mon[28203]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-17T00:15:25.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:25 smithi002 ceph-mon[28203]: osdmap e45: 7 total, 6 up, 7 in 2024-09-17T00:15:25.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:25 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T00:15:25.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:25 smithi002 ceph-mon[28203]: from='osd.6 [v2:172.21.15.99:6816/462153986,v1:172.21.15.99:6817/462153986]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi099", "root=default"]}]: dispatch 2024-09-17T00:15:25.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:25 smithi002 ceph-mon[28203]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi099", "root=default"]}]: dispatch 2024-09-17T00:15:26.906 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:26 smithi099 ceph-mon[24594]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi099", "root=default"]}]': finished 2024-09-17T00:15:26.906 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:26 smithi099 ceph-mon[24594]: osdmap e46: 7 total, 6 up, 7 in 2024-09-17T00:15:26.906 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:26 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T00:15:26.906 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:26 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T00:15:26.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:26 smithi002 ceph-mon[28203]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi099", "root=default"]}]': finished 2024-09-17T00:15:26.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:26 smithi002 ceph-mon[28203]: osdmap e46: 7 total, 6 up, 7 in 2024-09-17T00:15:26.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:26 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T00:15:26.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:26 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T00:15:26.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:26 smithi002 ceph-mon[23737]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi099", "root=default"]}]': finished 2024-09-17T00:15:26.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:26 smithi002 ceph-mon[23737]: osdmap e46: 7 total, 6 up, 7 in 2024-09-17T00:15:26.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:26 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T00:15:26.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:26 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T00:15:27.531 INFO:teuthology.orchestra.run.smithi099.stdout: 2024-09-17T00:15:27.552 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3aad27a2-7489-11ef-bceb-c7b262605968 -- ceph orch daemon add osd smithi099:vg_nvme/lv_1 2024-09-17T00:15:27.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:27 smithi099 ceph-mon[24594]: purged_snaps scrub starts 2024-09-17T00:15:27.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:27 smithi099 ceph-mon[24594]: purged_snaps scrub ok 2024-09-17T00:15:27.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:27 smithi099 ceph-mon[24594]: pgmap v147: 1 pgs: 1 peering; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-09-17T00:15:27.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:27 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T00:15:27.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:27 smithi099 ceph-mon[24594]: osd.6 [v2:172.21.15.99:6816/462153986,v1:172.21.15.99:6817/462153986] boot 2024-09-17T00:15:27.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:27 smithi099 ceph-mon[24594]: osdmap e47: 7 total, 7 up, 7 in 2024-09-17T00:15:27.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:27 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T00:15:27.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:27 smithi099 ceph-mon[24594]: osdmap e48: 7 total, 7 up, 7 in 2024-09-17T00:15:27.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:27 smithi002 ceph-mon[23737]: purged_snaps scrub starts 2024-09-17T00:15:27.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:27 smithi002 ceph-mon[23737]: purged_snaps scrub ok 2024-09-17T00:15:27.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:27 smithi002 ceph-mon[23737]: pgmap v147: 1 pgs: 1 peering; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-09-17T00:15:27.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:27 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T00:15:27.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:27 smithi002 ceph-mon[23737]: osd.6 [v2:172.21.15.99:6816/462153986,v1:172.21.15.99:6817/462153986] boot 2024-09-17T00:15:27.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:27 smithi002 ceph-mon[23737]: osdmap e47: 7 total, 7 up, 7 in 2024-09-17T00:15:27.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:27 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T00:15:27.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:27 smithi002 ceph-mon[23737]: osdmap e48: 7 total, 7 up, 7 in 2024-09-17T00:15:27.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:27 smithi002 ceph-mon[28203]: purged_snaps scrub starts 2024-09-17T00:15:27.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:27 smithi002 ceph-mon[28203]: purged_snaps scrub ok 2024-09-17T00:15:27.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:27 smithi002 ceph-mon[28203]: pgmap v147: 1 pgs: 1 peering; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-09-17T00:15:27.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:27 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T00:15:27.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:27 smithi002 ceph-mon[28203]: osd.6 [v2:172.21.15.99:6816/462153986,v1:172.21.15.99:6817/462153986] boot 2024-09-17T00:15:27.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:27 smithi002 ceph-mon[28203]: osdmap e47: 7 total, 7 up, 7 in 2024-09-17T00:15:27.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:27 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T00:15:27.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:27 smithi002 ceph-mon[28203]: osdmap e48: 7 total, 7 up, 7 in 2024-09-17T00:15:29.716 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:29 smithi099 ceph-mon[24594]: pgmap v150: 1 pgs: 1 peering; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-09-17T00:15:29.716 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:29 smithi099 ceph-mon[24594]: osdmap e49: 7 total, 7 up, 7 in 2024-09-17T00:15:29.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:29 smithi002 ceph-mon[23737]: pgmap v150: 1 pgs: 1 peering; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-09-17T00:15:29.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:29 smithi002 ceph-mon[23737]: osdmap e49: 7 total, 7 up, 7 in 2024-09-17T00:15:29.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:29 smithi002 ceph-mon[28203]: pgmap v150: 1 pgs: 1 peering; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-09-17T00:15:29.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:29 smithi002 ceph-mon[28203]: osdmap e49: 7 total, 7 up, 7 in 2024-09-17T00:15:30.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:30 smithi099 ceph-mon[24594]: osdmap e50: 7 total, 7 up, 7 in 2024-09-17T00:15:30.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:30 smithi099 ceph-mon[24594]: Detected new or changed devices on smithi099 2024-09-17T00:15:30.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:30 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:15:30.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:30 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-17T00:15:30.718 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:30 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-17T00:15:30.718 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:30 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-17T00:15:30.718 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:30 smithi099 ceph-mon[24594]: Adjusting osd_memory_target on smithi099 to 5696M 2024-09-17T00:15:30.718 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:30 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:15:30.718 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:30 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:15:30.718 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:30 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T00:15:30.718 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:30 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:15:30.718 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:30 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:15:30.718 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:30 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:15:30.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:30 smithi002 ceph-mon[23737]: osdmap e50: 7 total, 7 up, 7 in 2024-09-17T00:15:30.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:30 smithi002 ceph-mon[23737]: Detected new or changed devices on smithi099 2024-09-17T00:15:30.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:30 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:15:30.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:30 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-17T00:15:30.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:30 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-17T00:15:30.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:30 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-17T00:15:30.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:30 smithi002 ceph-mon[23737]: Adjusting osd_memory_target on smithi099 to 5696M 2024-09-17T00:15:30.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:30 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:15:30.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:30 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:15:30.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:30 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T00:15:30.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:30 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:15:30.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:30 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:15:30.981 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:30 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:15:30.981 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:30 smithi002 ceph-mon[28203]: osdmap e50: 7 total, 7 up, 7 in 2024-09-17T00:15:30.981 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:30 smithi002 ceph-mon[28203]: Detected new or changed devices on smithi099 2024-09-17T00:15:30.981 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:30 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:15:30.981 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:30 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-17T00:15:30.981 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:30 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-17T00:15:30.981 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:30 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-17T00:15:30.981 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:30 smithi002 ceph-mon[28203]: Adjusting osd_memory_target on smithi099 to 5696M 2024-09-17T00:15:30.981 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:30 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:15:30.981 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:30 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:15:30.982 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:30 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T00:15:30.982 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:30 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:15:30.982 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:30 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:15:30.982 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:30 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:15:31.966 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:31 smithi099 ceph-mon[24594]: pgmap v153: 1 pgs: 1 peering; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-09-17T00:15:31.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:31 smithi099 ceph-mon[24594]: osdmap e51: 7 total, 7 up, 7 in 2024-09-17T00:15:31.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:31 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-17T00:15:31.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:31 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-17T00:15:31.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:31 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:15:31.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:31 smithi002 ceph-mon[23737]: pgmap v153: 1 pgs: 1 peering; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-09-17T00:15:31.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:31 smithi002 ceph-mon[23737]: osdmap e51: 7 total, 7 up, 7 in 2024-09-17T00:15:31.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:31 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-17T00:15:31.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:31 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-17T00:15:31.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:31 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:15:31.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:31 smithi002 ceph-mon[28203]: pgmap v153: 1 pgs: 1 peering; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-09-17T00:15:31.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:31 smithi002 ceph-mon[28203]: osdmap e51: 7 total, 7 up, 7 in 2024-09-17T00:15:31.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:31 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-17T00:15:31.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:31 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-17T00:15:31.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:31 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:15:32.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:32 smithi099 ceph-mon[24594]: from='client.24299 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi099:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:15:32.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:32 smithi002 ceph-mon[23737]: from='client.24299 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi099:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:15:32.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:32 smithi002 ceph-mon[28203]: from='client.24299 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi099:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:15:33.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:33 smithi099 ceph-mon[24594]: pgmap v155: 1 pgs: 1 peering; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-09-17T00:15:33.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:33 smithi099 ceph-mon[24594]: from='client.? 172.21.15.99:0/1568502546' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9b87a926-31b4-4937-b425-b52e773f79e1"}]: dispatch 2024-09-17T00:15:33.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:33 smithi099 ceph-mon[24594]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9b87a926-31b4-4937-b425-b52e773f79e1"}]: dispatch 2024-09-17T00:15:33.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:33 smithi099 ceph-mon[24594]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9b87a926-31b4-4937-b425-b52e773f79e1"}]': finished 2024-09-17T00:15:33.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:33 smithi099 ceph-mon[24594]: osdmap e52: 8 total, 7 up, 8 in 2024-09-17T00:15:33.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:33 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T00:15:33.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:33 smithi099 ceph-mon[24594]: from='client.? 172.21.15.99:0/2080172169' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-17T00:15:33.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:33 smithi002 ceph-mon[23737]: pgmap v155: 1 pgs: 1 peering; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-09-17T00:15:33.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:33 smithi002 ceph-mon[23737]: from='client.? 172.21.15.99:0/1568502546' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9b87a926-31b4-4937-b425-b52e773f79e1"}]: dispatch 2024-09-17T00:15:33.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:33 smithi002 ceph-mon[23737]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9b87a926-31b4-4937-b425-b52e773f79e1"}]: dispatch 2024-09-17T00:15:33.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:33 smithi002 ceph-mon[23737]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9b87a926-31b4-4937-b425-b52e773f79e1"}]': finished 2024-09-17T00:15:33.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:33 smithi002 ceph-mon[23737]: osdmap e52: 8 total, 7 up, 8 in 2024-09-17T00:15:33.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:33 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T00:15:33.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:33 smithi002 ceph-mon[23737]: from='client.? 172.21.15.99:0/2080172169' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-17T00:15:33.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:33 smithi002 ceph-mon[28203]: pgmap v155: 1 pgs: 1 peering; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-09-17T00:15:33.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:33 smithi002 ceph-mon[28203]: from='client.? 172.21.15.99:0/1568502546' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9b87a926-31b4-4937-b425-b52e773f79e1"}]: dispatch 2024-09-17T00:15:33.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:33 smithi002 ceph-mon[28203]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9b87a926-31b4-4937-b425-b52e773f79e1"}]: dispatch 2024-09-17T00:15:33.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:33 smithi002 ceph-mon[28203]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9b87a926-31b4-4937-b425-b52e773f79e1"}]': finished 2024-09-17T00:15:33.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:33 smithi002 ceph-mon[28203]: osdmap e52: 8 total, 7 up, 8 in 2024-09-17T00:15:33.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:33 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T00:15:33.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:33 smithi002 ceph-mon[28203]: from='client.? 172.21.15.99:0/2080172169' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-17T00:15:35.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:35 smithi099 ceph-mon[24594]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 92 KiB/s, 0 objects/s recovering 2024-09-17T00:15:35.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:35 smithi002 ceph-mon[23737]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 92 KiB/s, 0 objects/s recovering 2024-09-17T00:15:35.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:35 smithi002 ceph-mon[28203]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 92 KiB/s, 0 objects/s recovering 2024-09-17T00:15:37.829 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:37 smithi099 ceph-mon[24594]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 78 KiB/s, 0 objects/s recovering 2024-09-17T00:15:37.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:37 smithi002 ceph-mon[23737]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 78 KiB/s, 0 objects/s recovering 2024-09-17T00:15:37.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:37 smithi002 ceph-mon[28203]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 78 KiB/s, 0 objects/s recovering 2024-09-17T00:15:39.889 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:39 smithi099 ceph-mon[24594]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-09-17T00:15:39.889 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:39 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-09-17T00:15:39.889 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:39 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:15:39.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:39 smithi002 ceph-mon[23737]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-09-17T00:15:39.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:39 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-09-17T00:15:39.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:39 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:15:39.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:39 smithi002 ceph-mon[28203]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-09-17T00:15:39.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:39 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-09-17T00:15:39.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:39 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:15:40.618 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:40 smithi099 ceph-mon[24594]: Deploying daemon osd.7 on smithi099 2024-09-17T00:15:40.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:40 smithi002 ceph-mon[23737]: Deploying daemon osd.7 on smithi099 2024-09-17T00:15:40.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:40 smithi002 ceph-mon[28203]: Deploying daemon osd.7 on smithi099 2024-09-17T00:15:41.758 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:41 smithi099 ceph-mon[24594]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 54 KiB/s, 0 objects/s recovering 2024-09-17T00:15:41.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:41 smithi002 ceph-mon[23737]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 54 KiB/s, 0 objects/s recovering 2024-09-17T00:15:41.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:41 smithi002 ceph-mon[28203]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 54 KiB/s, 0 objects/s recovering 2024-09-17T00:15:43.683 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:43 smithi099 ceph-mon[24594]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-09-17T00:15:43.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:43 smithi002 ceph-mon[23737]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-09-17T00:15:43.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:43 smithi002 ceph-mon[28203]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-09-17T00:15:44.898 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:44 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T00:15:44.898 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:44 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T00:15:44.898 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:44 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T00:15:44.898 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:44 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:15:44.898 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:44 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:15:44.898 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:44 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:15:44.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:44 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T00:15:44.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:44 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T00:15:44.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:44 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T00:15:44.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:44 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:15:44.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:44 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:15:44.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:44 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:15:44.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:44 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T00:15:44.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:44 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T00:15:44.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:44 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T00:15:44.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:44 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:15:44.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:44 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:15:44.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:44 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:15:45.780 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:45 smithi099 ceph-mon[24594]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-09-17T00:15:45.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:45 smithi002 ceph-mon[23737]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-09-17T00:15:45.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:45 smithi002 ceph-mon[28203]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-09-17T00:15:47.237 INFO:teuthology.orchestra.run.smithi099.stdout:Created osd(s) 7 on host 'smithi099' 2024-09-17T00:15:47.900 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:47 smithi099 ceph-mon[24594]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-09-17T00:15:47.900 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:47 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:15:47.900 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:47 smithi099 ceph-mon[24594]: from='osd.7 [v2:172.21.15.99:6824/544554822,v1:172.21.15.99:6825/544554822]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-17T00:15:47.900 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:47 smithi099 ceph-mon[24594]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-17T00:15:47.922 DEBUG:teuthology.orchestra.run.smithi099:osd.7> sudo journalctl -f -n 0 -u ceph-3aad27a2-7489-11ef-bceb-c7b262605968@osd.7.service 2024-09-17T00:15:47.925 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2024-09-17T00:15:47.925 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3aad27a2-7489-11ef-bceb-c7b262605968 -- ceph osd stat -f json 2024-09-17T00:15:47.953 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:47 smithi002 ceph-mon[23737]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-09-17T00:15:47.953 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:47 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:15:47.953 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:47 smithi002 ceph-mon[23737]: from='osd.7 [v2:172.21.15.99:6824/544554822,v1:172.21.15.99:6825/544554822]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-17T00:15:47.953 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:47 smithi002 ceph-mon[23737]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-17T00:15:47.954 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:47 smithi002 ceph-mon[28203]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-09-17T00:15:47.954 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:47 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:15:47.954 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:47 smithi002 ceph-mon[28203]: from='osd.7 [v2:172.21.15.99:6824/544554822,v1:172.21.15.99:6825/544554822]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-17T00:15:47.954 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:47 smithi002 ceph-mon[28203]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-17T00:15:49.355 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:49 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:15:49.355 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:49 smithi002 ceph-mon[23737]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-09-17T00:15:49.355 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:49 smithi002 ceph-mon[23737]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-17T00:15:49.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:49 smithi002 ceph-mon[23737]: osdmap e53: 8 total, 7 up, 8 in 2024-09-17T00:15:49.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:49 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T00:15:49.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:49 smithi002 ceph-mon[23737]: from='osd.7 [v2:172.21.15.99:6824/544554822,v1:172.21.15.99:6825/544554822]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi099", "root=default"]}]: dispatch 2024-09-17T00:15:49.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:49 smithi002 ceph-mon[23737]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi099", "root=default"]}]: dispatch 2024-09-17T00:15:49.356 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:49 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:15:49.356 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:49 smithi002 ceph-mon[28203]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-09-17T00:15:49.356 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:49 smithi002 ceph-mon[28203]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-17T00:15:49.356 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:49 smithi002 ceph-mon[28203]: osdmap e53: 8 total, 7 up, 8 in 2024-09-17T00:15:49.356 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:49 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T00:15:49.356 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:49 smithi002 ceph-mon[28203]: from='osd.7 [v2:172.21.15.99:6824/544554822,v1:172.21.15.99:6825/544554822]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi099", "root=default"]}]: dispatch 2024-09-17T00:15:49.357 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:49 smithi002 ceph-mon[28203]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi099", "root=default"]}]: dispatch 2024-09-17T00:15:49.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:49 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:15:49.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:49 smithi099 ceph-mon[24594]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-09-17T00:15:49.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:49 smithi099 ceph-mon[24594]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-17T00:15:49.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:49 smithi099 ceph-mon[24594]: osdmap e53: 8 total, 7 up, 8 in 2024-09-17T00:15:49.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:49 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T00:15:49.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:49 smithi099 ceph-mon[24594]: from='osd.7 [v2:172.21.15.99:6824/544554822,v1:172.21.15.99:6825/544554822]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi099", "root=default"]}]: dispatch 2024-09-17T00:15:49.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:49 smithi099 ceph-mon[24594]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi099", "root=default"]}]: dispatch 2024-09-17T00:15:49.468 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:15:49 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-7[36631]: 2024-09-17T00:15:49.244+0000 7efc9b7d9700 -1 osd.7 0 waiting for initial osdmap 2024-09-17T00:15:49.468 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:15:49 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-7[36631]: 2024-09-17T00:15:49.251+0000 7efc9596f700 -1 osd.7 54 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T00:15:50.026 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-17T00:15:50.711 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:50 smithi002 ceph-mon[23737]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi099", "root=default"]}]': finished 2024-09-17T00:15:50.711 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:50 smithi002 ceph-mon[23737]: osdmap e54: 8 total, 7 up, 8 in 2024-09-17T00:15:50.711 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:50 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T00:15:50.712 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:50 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T00:15:50.712 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:50 smithi002 ceph-mon[23737]: from='client.? 172.21.15.2:0/1705603165' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-17T00:15:50.712 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:50 smithi002 ceph-mon[28203]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi099", "root=default"]}]': finished 2024-09-17T00:15:50.712 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:50 smithi002 ceph-mon[28203]: osdmap e54: 8 total, 7 up, 8 in 2024-09-17T00:15:50.712 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:50 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T00:15:50.712 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:50 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T00:15:50.712 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:50 smithi002 ceph-mon[28203]: from='client.? 172.21.15.2:0/1705603165' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-17T00:15:50.716 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:50 smithi099 ceph-mon[24594]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi099", "root=default"]}]': finished 2024-09-17T00:15:50.716 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:50 smithi099 ceph-mon[24594]: osdmap e54: 8 total, 7 up, 8 in 2024-09-17T00:15:50.716 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:50 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T00:15:50.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:50 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T00:15:50.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:50 smithi099 ceph-mon[24594]: from='client.? 172.21.15.2:0/1705603165' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-17T00:15:50.740 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":54,"num_osds":8,"num_up_osds":7,"osd_up_since":1726532126,"num_in_osds":8,"osd_in_since":1726532132,"num_remapped_pgs":0} 2024-09-17T00:15:51.399 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:51 smithi099 ceph-mon[24594]: purged_snaps scrub starts 2024-09-17T00:15:51.399 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:51 smithi099 ceph-mon[24594]: purged_snaps scrub ok 2024-09-17T00:15:51.399 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:51 smithi099 ceph-mon[24594]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-09-17T00:15:51.399 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:51 smithi099 ceph-mon[24594]: osd.7 [v2:172.21.15.99:6824/544554822,v1:172.21.15.99:6825/544554822] boot 2024-09-17T00:15:51.399 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:51 smithi099 ceph-mon[24594]: osdmap e55: 8 total, 8 up, 8 in 2024-09-17T00:15:51.399 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:51 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T00:15:51.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:51 smithi002 ceph-mon[23737]: purged_snaps scrub starts 2024-09-17T00:15:51.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:51 smithi002 ceph-mon[23737]: purged_snaps scrub ok 2024-09-17T00:15:51.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:51 smithi002 ceph-mon[23737]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-09-17T00:15:51.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:51 smithi002 ceph-mon[23737]: osd.7 [v2:172.21.15.99:6824/544554822,v1:172.21.15.99:6825/544554822] boot 2024-09-17T00:15:51.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:51 smithi002 ceph-mon[23737]: osdmap e55: 8 total, 8 up, 8 in 2024-09-17T00:15:51.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:51 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T00:15:51.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:51 smithi002 ceph-mon[28203]: purged_snaps scrub starts 2024-09-17T00:15:51.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:51 smithi002 ceph-mon[28203]: purged_snaps scrub ok 2024-09-17T00:15:51.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:51 smithi002 ceph-mon[28203]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-09-17T00:15:51.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:51 smithi002 ceph-mon[28203]: osd.7 [v2:172.21.15.99:6824/544554822,v1:172.21.15.99:6825/544554822] boot 2024-09-17T00:15:51.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:51 smithi002 ceph-mon[28203]: osdmap e55: 8 total, 8 up, 8 in 2024-09-17T00:15:51.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:51 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T00:15:51.742 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3aad27a2-7489-11ef-bceb-c7b262605968 -- ceph osd stat -f json 2024-09-17T00:15:52.314 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:52 smithi002 ceph-mon[23737]: osdmap e56: 8 total, 8 up, 8 in 2024-09-17T00:15:52.314 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:52 smithi002 ceph-mon[23737]: Detected new or changed devices on smithi099 2024-09-17T00:15:52.314 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:52 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:15:52.314 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:52 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-17T00:15:52.315 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:52 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-17T00:15:52.315 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:52 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-17T00:15:52.315 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:52 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-17T00:15:52.315 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:52 smithi002 ceph-mon[23737]: Adjusting osd_memory_target on smithi099 to 4272M 2024-09-17T00:15:52.315 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:52 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:15:52.315 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:52 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:15:52.315 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:52 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T00:15:52.315 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:52 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:15:52.315 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:52 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:15:52.315 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:52 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:15:52.315 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:52 smithi002 ceph-mon[28203]: osdmap e56: 8 total, 8 up, 8 in 2024-09-17T00:15:52.316 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:52 smithi002 ceph-mon[28203]: Detected new or changed devices on smithi099 2024-09-17T00:15:52.316 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:52 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:15:52.316 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:52 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-17T00:15:52.316 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:52 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-17T00:15:52.316 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:52 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-17T00:15:52.316 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:52 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-17T00:15:52.316 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:52 smithi002 ceph-mon[28203]: Adjusting osd_memory_target on smithi099 to 4272M 2024-09-17T00:15:52.316 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:52 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:15:52.316 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:52 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:15:52.316 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:52 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T00:15:52.316 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:52 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:15:52.316 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:52 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:15:52.316 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:52 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:15:52.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:52 smithi099 ceph-mon[24594]: osdmap e56: 8 total, 8 up, 8 in 2024-09-17T00:15:52.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:52 smithi099 ceph-mon[24594]: Detected new or changed devices on smithi099 2024-09-17T00:15:52.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:52 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:15:52.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:52 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-17T00:15:52.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:52 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-17T00:15:52.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:52 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-17T00:15:52.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:52 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-17T00:15:52.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:52 smithi099 ceph-mon[24594]: Adjusting osd_memory_target on smithi099 to 4272M 2024-09-17T00:15:52.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:52 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:15:52.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:52 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:15:52.718 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:52 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T00:15:52.718 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:52 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:15:52.718 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:52 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:15:52.718 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:52 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:15:53.317 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:53 smithi002 ceph-mon[28203]: pgmap v170: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:15:53.317 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:53 smithi002 ceph-mon[28203]: osdmap e57: 8 total, 8 up, 8 in 2024-09-17T00:15:53.317 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:53 smithi002 ceph-mon[28203]: osdmap e58: 8 total, 8 up, 8 in 2024-09-17T00:15:53.318 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:53 smithi002 ceph-mon[23737]: pgmap v170: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:15:53.318 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:53 smithi002 ceph-mon[23737]: osdmap e57: 8 total, 8 up, 8 in 2024-09-17T00:15:53.318 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:53 smithi002 ceph-mon[23737]: osdmap e58: 8 total, 8 up, 8 in 2024-09-17T00:15:53.716 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:53 smithi099 ceph-mon[24594]: pgmap v170: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:15:53.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:53 smithi099 ceph-mon[24594]: osdmap e57: 8 total, 8 up, 8 in 2024-09-17T00:15:53.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:53 smithi099 ceph-mon[24594]: osdmap e58: 8 total, 8 up, 8 in 2024-09-17T00:15:53.743 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-17T00:15:54.654 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":59,"num_osds":8,"num_up_osds":8,"osd_up_since":1726532150,"num_in_osds":8,"osd_in_since":1726532132,"num_remapped_pgs":0} 2024-09-17T00:15:54.655 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 3aad27a2-7489-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-17T00:15:54.875 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T00:15:54.966 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:54 smithi099 ceph-mon[24594]: osdmap e59: 8 total, 8 up, 8 in 2024-09-17T00:15:54.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:54 smithi099 ceph-mon[24594]: from='client.? 172.21.15.2:0/4005338353' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-17T00:15:54.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:54 smithi002 ceph-mon[28203]: osdmap e59: 8 total, 8 up, 8 in 2024-09-17T00:15:54.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:54 smithi002 ceph-mon[28203]: from='client.? 172.21.15.2:0/4005338353' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-17T00:15:54.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:54 smithi002 ceph-mon[23737]: osdmap e59: 8 total, 8 up, 8 in 2024-09-17T00:15:54.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:54 smithi002 ceph-mon[23737]: from='client.? 172.21.15.2:0/4005338353' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-17T00:15:55.966 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:55 smithi099 ceph-mon[24594]: pgmap v174: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:15:55.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:55 smithi002 ceph-mon[23737]: pgmap v174: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:15:55.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:55 smithi002 ceph-mon[28203]: pgmap v174: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:15:56.997 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-17T00:15:56.997 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":59,"fsid":"3aad27a2-7489-11ef-bceb-c7b262605968","created":"2024-09-17T00:10:48.442986+0000","modified":"2024-09-17T00:15:53.549978+0000","last_up_change":"2024-09-17T00:15:50.239941+0000","last_in_change":"2024-09-17T00:15:32.949068+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T00:13:54.650037+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"2030f6f2-3659-4b78-bca5-b74d15218c34","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":2656745758},{"type":"v1","addr":"172.21.15.2:6803","nonce":2656745758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":2656745758},{"type":"v1","addr":"172.21.15.2:6805","nonce":2656745758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":2656745758},{"type":"v1","addr":"172.21.15.2:6809","nonce":2656745758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":2656745758},{"type":"v1","addr":"172.21.15.2:6807","nonce":2656745758}]},"public_addr":"172.21.15.2:6803/2656745758","cluster_addr":"172.21.15.2:6805/2656745758","heartbeat_back_addr":"172.21.15.2:6809/2656745758","heartbeat_front_addr":"172.21.15.2:6807/2656745758","state":["exists","up"]},{"osd":1,"uuid":"e3883c47-fd8d-444f-8057-25d08b59af31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":981024217},{"type":"v1","addr":"172.21.15.2:6811","nonce":981024217}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":981024217},{"type":"v1","addr":"172.21.15.2:6813","nonce":981024217}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":981024217},{"type":"v1","addr":"172.21.15.2:6817","nonce":981024217}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":981024217},{"type":"v1","addr":"172.21.15.2:6815","nonce":981024217}]},"public_addr":"172.21.15.2:6811/981024217","cluster_addr":"172.21.15.2:6813/981024217","heartbeat_back_addr":"172.21.15.2:6817/981024217","heartbeat_front_addr":"172.21.15.2:6815/981024217","state":["exists","up"]},{"osd":2,"uuid":"9d042533-63e5-4f38-a8db-10b61e9b39d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":2219840596},{"type":"v1","addr":"172.21.15.2:6819","nonce":2219840596}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":2219840596},{"type":"v1","addr":"172.21.15.2:6821","nonce":2219840596}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":2219840596},{"type":"v1","addr":"172.21.15.2:6825","nonce":2219840596}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":2219840596},{"type":"v1","addr":"172.21.15.2:6823","nonce":2219840596}]},"public_addr":"172.21.15.2:6819/2219840596","cluster_addr":"172.21.15.2:6821/2219840596","heartbeat_back_addr":"172.21.15.2:6825/2219840596","heartbeat_front_addr":"172.21.15.2:6823/2219840596","state":["exists","up"]},{"osd":3,"uuid":"d9475026-f3ce-4b45-b5de-780c78dbc233","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6826","nonce":1628413766},{"type":"v1","addr":"172.21.15.2:6827","nonce":1628413766}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":1628413766},{"type":"v1","addr":"172.21.15.2:6829","nonce":1628413766}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":1628413766},{"type":"v1","addr":"172.21.15.2:6833","nonce":1628413766}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":1628413766},{"type":"v1","addr":"172.21.15.2:6831","nonce":1628413766}]},"public_addr":"172.21.15.2:6827/1628413766","cluster_addr":"172.21.15.2:6829/1628413766","heartbeat_back_addr":"172.21.15.2:6833/1628413766","heartbeat_front_addr":"172.21.15.2:6831/1628413766","state":["exists","up"]},{"osd":4,"uuid":"f8f0575a-3699-4e8f-b969-e4b6973708df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6800","nonce":2601657006},{"type":"v1","addr":"172.21.15.99:6801","nonce":2601657006}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6802","nonce":2601657006},{"type":"v1","addr":"172.21.15.99:6803","nonce":2601657006}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6806","nonce":2601657006},{"type":"v1","addr":"172.21.15.99:6807","nonce":2601657006}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6804","nonce":2601657006},{"type":"v1","addr":"172.21.15.99:6805","nonce":2601657006}]},"public_addr":"172.21.15.99:6801/2601657006","cluster_addr":"172.21.15.99:6803/2601657006","heartbeat_back_addr":"172.21.15.99:6807/2601657006","heartbeat_front_addr":"172.21.15.99:6805/2601657006","state":["exists","up"]},{"osd":5,"uuid":"c0ba894b-bcbd-40e5-b00d-73dbddc87d59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6808","nonce":1930951601},{"type":"v1","addr":"172.21.15.99:6809","nonce":1930951601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6810","nonce":1930951601},{"type":"v1","addr":"172.21.15.99:6811","nonce":1930951601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6814","nonce":1930951601},{"type":"v1","addr":"172.21.15.99:6815","nonce":1930951601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6812","nonce":1930951601},{"type":"v1","addr":"172.21.15.99:6813","nonce":1930951601}]},"public_addr":"172.21.15.99:6809/1930951601","cluster_addr":"172.21.15.99:6811/1930951601","heartbeat_back_addr":"172.21.15.99:6815/1930951601","heartbeat_front_addr":"172.21.15.99:6813/1930951601","state":["exists","up"]},{"osd":6,"uuid":"9aeb407a-4fd9-415c-aa66-1e66d8aaa700","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6816","nonce":462153986},{"type":"v1","addr":"172.21.15.99:6817","nonce":462153986}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6818","nonce":462153986},{"type":"v1","addr":"172.21.15.99:6819","nonce":462153986}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6822","nonce":462153986},{"type":"v1","addr":"172.21.15.99:6823","nonce":462153986}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6820","nonce":462153986},{"type":"v1","addr":"172.21.15.99:6821","nonce":462153986}]},"public_addr":"172.21.15.99:6817/462153986","cluster_addr":"172.21.15.99:6819/462153986","heartbeat_back_addr":"172.21.15.99:6823/462153986","heartbeat_front_addr":"172.21.15.99:6821/462153986","state":["exists","up"]},{"osd":7,"uuid":"9b87a926-31b4-4937-b425-b52e773f79e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6824","nonce":544554822},{"type":"v1","addr":"172.21.15.99:6825","nonce":544554822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6826","nonce":544554822},{"type":"v1","addr":"172.21.15.99:6827","nonce":544554822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6830","nonce":544554822},{"type":"v1","addr":"172.21.15.99:6831","nonce":544554822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6828","nonce":544554822},{"type":"v1","addr":"172.21.15.99:6829","nonce":544554822}]},"public_addr":"172.21.15.99:6825/544554822","cluster_addr":"172.21.15.99:6827/544554822","heartbeat_back_addr":"172.21.15.99:6831/544554822","heartbeat_front_addr":"172.21.15.99:6829/544554822","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T00:13:02.874193+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T00:13:28.064593+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T00:13:51.522483+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T00:14:15.866757+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T00:14:37.409107+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T00:15:00.424239+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T00:15:24.796694+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T00:15:48.372673+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.2:0/1780251331":"2024-09-18T00:11:44.094172+0000","172.21.15.2:0/824729147":"2024-09-18T00:11:44.094172+0000","172.21.15.2:6801/2522360616":"2024-09-18T00:11:44.094172+0000","172.21.15.2:6800/2522360616":"2024-09-18T00:11:44.094172+0000","172.21.15.2:0/3091500665":"2024-09-18T00:11:44.094172+0000","172.21.15.2:0/3536227112":"2024-09-18T00:11:20.938701+0000","172.21.15.2:6800/3667297605":"2024-09-18T00:11:20.938701+0000","172.21.15.2:6801/3667297605":"2024-09-18T00:11:20.938701+0000","172.21.15.2:0/3867827864":"2024-09-18T00:11:20.938701+0000","172.21.15.2:0/174032459":"2024-09-18T00:11:20.938701+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T00:15:57.624 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-17T00:13:54.650037+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2024-09-17T00:15:57.624 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 3aad27a2-7489-11ef-bceb-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-09-17T00:15:57.841 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T00:15:57.962 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:57 smithi002 ceph-mon[23737]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 104 KiB/s, 0 objects/s recovering 2024-09-17T00:15:57.962 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:57 smithi002 ceph-mon[23737]: from='client.? 172.21.15.2:0/3785698708' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-17T00:15:57.963 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:57 smithi002 ceph-mon[28203]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 104 KiB/s, 0 objects/s recovering 2024-09-17T00:15:57.963 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:57 smithi002 ceph-mon[28203]: from='client.? 172.21.15.2:0/3785698708' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-17T00:15:57.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:57 smithi099 ceph-mon[24594]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 104 KiB/s, 0 objects/s recovering 2024-09-17T00:15:57.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:57 smithi099 ceph-mon[24594]: from='client.? 172.21.15.2:0/3785698708' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-17T00:15:59.661 INFO:teuthology.orchestra.run.smithi002.stdout:pg_num: 1 2024-09-17T00:15:59.661 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:15:59 smithi002 ceph-mon[23737]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-09-17T00:15:59.661 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:15:59 smithi002 ceph-mon[28203]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-09-17T00:15:59.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:15:59 smithi099 ceph-mon[24594]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-09-17T00:16:00.318 INFO:tasks.cephadm:Adding prometheus.a on smithi099 2024-09-17T00:16:00.319 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3aad27a2-7489-11ef-bceb-c7b262605968 -- ceph orch apply prometheus '1;smithi099=a' 2024-09-17T00:16:00.630 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:16:00 smithi099 ceph-mon[24594]: from='client.? 172.21.15.2:0/2427258128' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-17T00:16:00.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:16:00 smithi002 ceph-mon[23737]: from='client.? 172.21.15.2:0/2427258128' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-17T00:16:00.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:16:00 smithi002 ceph-mon[28203]: from='client.? 172.21.15.2:0/2427258128' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-17T00:16:01.781 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:16:01 smithi099 ceph-mon[24594]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-09-17T00:16:01.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:16:01 smithi002 ceph-mon[28203]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-09-17T00:16:01.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:16:01 smithi002 ceph-mon[23737]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-09-17T00:16:02.168 INFO:teuthology.orchestra.run.smithi099.stdout:Scheduled prometheus update... 2024-09-17T00:16:02.860 DEBUG:teuthology.orchestra.run.smithi099:prometheus.a> sudo journalctl -f -n 0 -u ceph-3aad27a2-7489-11ef-bceb-c7b262605968@prometheus.a.service 2024-09-17T00:16:02.862 INFO:tasks.cephadm:Adding node-exporter.a on smithi002 2024-09-17T00:16:02.862 INFO:tasks.cephadm:Adding node-exporter.b on smithi099 2024-09-17T00:16:02.863 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3aad27a2-7489-11ef-bceb-c7b262605968 -- ceph orch apply node-exporter '2;smithi002=a;smithi099=b' 2024-09-17T00:16:03.187 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:16:03 smithi099 ceph-mon[24594]: from='client.24350 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi099=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:16:03.187 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:16:03 smithi099 ceph-mon[24594]: Saving service prometheus spec with placement smithi099=a;count:1 2024-09-17T00:16:03.188 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:16:03 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:16:03.188 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:16:03 smithi099 ceph-mon[24594]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s, 0 objects/s recovering 2024-09-17T00:16:03.188 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:16:03 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T00:16:03.188 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:16:03 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:16:03.188 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:16:03 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:16:03.188 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:16:03 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:16:03.188 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:16:03 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-09-17T00:16:03.441 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:16:03 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[25772]: ignoring --setuser ceph since I am not root 2024-09-17T00:16:03.442 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:16:03 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[25772]: ignoring --setgroup ceph since I am not root 2024-09-17T00:16:03.442 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:16:03 smithi099 ceph-mgr[25776]: -- 172.21.15.99:0/1564421802 <== mon.2 v2:172.21.15.99:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5609c9e10340 con 0x5609c9de8800 2024-09-17T00:16:03.442 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:16:03 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[25772]: 2024-09-17T00:16:03.362+0000 7f396a425000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-17T00:16:03.442 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:16:03 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[25772]: 2024-09-17T00:16:03.444+0000 7f396a425000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-17T00:16:03.456 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:16:03 smithi002 ceph-mon[23737]: from='client.24350 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi099=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:16:03.456 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:16:03 smithi002 ceph-mon[23737]: Saving service prometheus spec with placement smithi099=a;count:1 2024-09-17T00:16:03.457 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:16:03 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:16:03.457 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:16:03 smithi002 ceph-mon[23737]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s, 0 objects/s recovering 2024-09-17T00:16:03.457 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:16:03 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T00:16:03.457 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:16:03 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:16:03.457 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:16:03 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:16:03.457 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:16:03 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:16:03.458 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:16:03 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-09-17T00:16:03.458 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:16:03 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: ignoring --setuser ceph since I am not root 2024-09-17T00:16:03.458 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:16:03 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: ignoring --setgroup ceph since I am not root 2024-09-17T00:16:03.458 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:16:03 smithi002 ceph-mgr[23968]: -- 172.21.15.2:0/2852805159 <== mon.1 v2:172.21.15.2:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x56250eafa340 con 0x56250ead3000 2024-09-17T00:16:03.459 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:16:03 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: 2024-09-17T00:16:03.377+0000 7f597402f000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-17T00:16:03.459 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:16:03 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: 2024-09-17T00:16:03.458+0000 7f597402f000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-17T00:16:03.459 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:16:03 smithi002 ceph-mon[28203]: from='client.24350 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi099=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:16:03.459 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:16:03 smithi002 ceph-mon[28203]: Saving service prometheus spec with placement smithi099=a;count:1 2024-09-17T00:16:03.460 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:16:03 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:16:03.460 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:16:03 smithi002 ceph-mon[28203]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s, 0 objects/s recovering 2024-09-17T00:16:03.460 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:16:03 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T00:16:03.460 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:16:03 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:16:03.460 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:16:03 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:16:03.461 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:16:03 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' 2024-09-17T00:16:03.461 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:16:03 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-09-17T00:16:03.704 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:16:03 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[25772]: 2024-09-17T00:16:03.619+0000 7f396a425000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-17T00:16:03.729 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:16:03 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: 2024-09-17T00:16:03.650+0000 7f597402f000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-17T00:16:03.966 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:16:03 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[25772]: 2024-09-17T00:16:03.706+0000 7f396a425000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-17T00:16:03.967 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:16:03 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[25772]: 2024-09-17T00:16:03.840+0000 7f396a425000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-17T00:16:04.091 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:16:03 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: 2024-09-17T00:16:03.741+0000 7f597402f000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-17T00:16:04.092 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:16:03 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: 2024-09-17T00:16:03.877+0000 7f597402f000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-17T00:16:04.092 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:16:04 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: 2024-09-17T00:16:04.094+0000 7f597402f000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-17T00:16:04.385 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:16:04 smithi099 ceph-mon[24594]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-09-17T00:16:04.385 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:16:04 smithi099 ceph-mon[24594]: mgrmap e16: y(active, since 4m), standbys: x 2024-09-17T00:16:04.386 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:16:04 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[25772]: 2024-09-17T00:16:04.053+0000 7f396a425000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-17T00:16:04.435 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:16:04 smithi002 ceph-mon[23737]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-09-17T00:16:04.435 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:16:04 smithi002 ceph-mon[23737]: mgrmap e16: y(active, since 4m), standbys: x 2024-09-17T00:16:04.435 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:16:04 smithi002 ceph-mon[28203]: from='mgr.14152 172.21.15.2:0/2315053031' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-09-17T00:16:04.435 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:16:04 smithi002 ceph-mon[28203]: mgrmap e16: y(active, since 4m), standbys: x 2024-09-17T00:16:04.649 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:16:04 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[25772]: 2024-09-17T00:16:04.388+0000 7f396a425000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-17T00:16:04.649 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:16:04 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[25772]: 2024-09-17T00:16:04.478+0000 7f396a425000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-17T00:16:04.649 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:16:04 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[25772]: 2024-09-17T00:16:04.651+0000 7f396a425000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-17T00:16:04.699 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:16:04 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: 2024-09-17T00:16:04.437+0000 7f597402f000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-17T00:16:04.699 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:16:04 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: 2024-09-17T00:16:04.529+0000 7f597402f000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-17T00:16:04.699 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:16:04 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: 2024-09-17T00:16:04.701+0000 7f597402f000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-17T00:16:06.124 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:16:05 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: 2024-09-17T00:16:05.800+0000 7f597402f000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-17T00:16:06.125 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:16:05 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: 2024-09-17T00:16:05.897+0000 7f597402f000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-17T00:16:06.217 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:16:05 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[25772]: 2024-09-17T00:16:05.720+0000 7f396a425000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-17T00:16:06.217 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:16:05 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[25772]: 2024-09-17T00:16:05.810+0000 7f396a425000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-17T00:16:06.967 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:16:06 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[25772]: 2024-09-17T00:16:06.698+0000 7f396a425000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-17T00:16:07.118 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:16:06 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: 2024-09-17T00:16:06.864+0000 7f597402f000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-17T00:16:07.320 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:16:07 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[25772]: 2024-09-17T00:16:07.005+0000 7f396a425000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-17T00:16:07.321 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:16:07 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[25772]: 2024-09-17T00:16:07.139+0000 7f396a425000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-17T00:16:07.321 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:16:07 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[25772]: 2024-09-17T00:16:07.227+0000 7f396a425000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-17T00:16:07.422 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:16:07 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: 2024-09-17T00:16:07.191+0000 7f597402f000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-17T00:16:07.422 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:16:07 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: 2024-09-17T00:16:07.331+0000 7f597402f000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-17T00:16:07.648 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:16:07 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[25772]: 2024-09-17T00:16:07.323+0000 7f396a425000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-17T00:16:07.648 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:16:07 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[25772]: 2024-09-17T00:16:07.411+0000 7f396a425000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-17T00:16:07.648 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:16:07 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[25772]: 2024-09-17T00:16:07.557+0000 7f396a425000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-17T00:16:07.729 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:16:07 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: 2024-09-17T00:16:07.425+0000 7f597402f000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-17T00:16:07.729 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:16:07 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: 2024-09-17T00:16:07.528+0000 7f597402f000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-17T00:16:07.729 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:16:07 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: 2024-09-17T00:16:07.621+0000 7f597402f000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-17T00:16:07.966 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:16:07 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[25772]: 2024-09-17T00:16:07.651+0000 7f396a425000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-17T00:16:07.967 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:16:07 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[25772]: 2024-09-17T00:16:07.747+0000 7f396a425000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-17T00:16:08.229 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:16:07 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: 2024-09-17T00:16:07.777+0000 7f597402f000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-17T00:16:08.229 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:16:07 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: 2024-09-17T00:16:07.875+0000 7f597402f000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-17T00:16:08.229 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:16:07 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: 2024-09-17T00:16:07.976+0000 7f597402f000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-17T00:16:08.466 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:16:08 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[25772]: 2024-09-17T00:16:08.030+0000 7f396a425000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-17T00:16:08.467 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:16:08 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[25772]: 2024-09-17T00:16:08.119+0000 7f396a425000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-17T00:16:08.467 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:16:08 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[25772]: 2024-09-17T00:16:08.208+0000 7f396a425000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-17T00:16:08.729 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:16:08 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: 2024-09-17T00:16:08.273+0000 7f597402f000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-17T00:16:08.729 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:16:08 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: 2024-09-17T00:16:08.366+0000 7f597402f000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-17T00:16:08.729 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:16:08 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: 2024-09-17T00:16:08.460+0000 7f597402f000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-17T00:16:09.582 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:16:09 smithi002 ceph-mon[23737]: Standby manager daemon x restarted 2024-09-17T00:16:09.582 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:16:09 smithi002 ceph-mon[23737]: Standby manager daemon x started 2024-09-17T00:16:09.582 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:16:09 smithi002 ceph-mon[23737]: from='mgr.? 172.21.15.99:0/4232943395' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-17T00:16:09.582 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:16:09 smithi002 ceph-mon[23737]: from='mgr.? 172.21.15.99:0/4232943395' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-17T00:16:09.582 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:16:09 smithi002 ceph-mon[23737]: from='mgr.? 172.21.15.99:0/4232943395' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-17T00:16:09.582 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:16:09 smithi002 ceph-mon[23737]: from='mgr.? 172.21.15.99:0/4232943395' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-17T00:16:09.582 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:16:09 smithi002 ceph-mon[28203]: Standby manager daemon x restarted 2024-09-17T00:16:09.582 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:16:09 smithi002 ceph-mon[28203]: Standby manager daemon x started 2024-09-17T00:16:09.583 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:16:09 smithi002 ceph-mon[28203]: from='mgr.? 172.21.15.99:0/4232943395' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-17T00:16:09.583 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:16:09 smithi002 ceph-mon[28203]: from='mgr.? 172.21.15.99:0/4232943395' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-17T00:16:09.583 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:16:09 smithi002 ceph-mon[28203]: from='mgr.? 172.21.15.99:0/4232943395' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-17T00:16:09.583 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:16:09 smithi002 ceph-mon[28203]: from='mgr.? 172.21.15.99:0/4232943395' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-17T00:16:09.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:16:09 smithi099 ceph-mon[24594]: Standby manager daemon x restarted 2024-09-17T00:16:09.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:16:09 smithi099 ceph-mon[24594]: Standby manager daemon x started 2024-09-17T00:16:09.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:16:09 smithi099 ceph-mon[24594]: from='mgr.? 172.21.15.99:0/4232943395' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-17T00:16:09.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:16:09 smithi099 ceph-mon[24594]: from='mgr.? 172.21.15.99:0/4232943395' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-17T00:16:09.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:16:09 smithi099 ceph-mon[24594]: from='mgr.? 172.21.15.99:0/4232943395' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-17T00:16:09.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:16:09 smithi099 ceph-mon[24594]: from='mgr.? 172.21.15.99:0/4232943395' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-17T00:16:09.718 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:16:09 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[25772]: 2024-09-17T00:16:09.271+0000 7f396a425000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-17T00:16:09.718 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:16:09 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[25772]: [17/Sep/2024:00:16:09] ENGINE Bus STARTING 2024-09-17T00:16:09.718 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:16:09 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[25772]: CherryPy Checker: 2024-09-17T00:16:09.718 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:16:09 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[25772]: The Application mounted at '' has an empty config. 2024-09-17T00:16:09.718 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:16:09 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[25772]: 2024-09-17T00:16:09.718 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:16:09 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[25772]: [17/Sep/2024:00:16:09] ENGINE Serving on http://:::9283 2024-09-17T00:16:09.718 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:16:09 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[25772]: [17/Sep/2024:00:16:09] ENGINE Bus STARTED 2024-09-17T00:16:09.979 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:16:09 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: 2024-09-17T00:16:09.584+0000 7f597402f000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-17T00:16:10.650 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:16:10 smithi099 ceph-mon[24594]: mgrmap e17: y(active, since 4m), standbys: x 2024-09-17T00:16:10.650 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:16:10 smithi099 ceph-mon[24594]: Active manager daemon y restarted 2024-09-17T00:16:10.650 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:16:10 smithi099 ceph-mon[24594]: Activating manager daemon y 2024-09-17T00:16:10.650 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:16:10 smithi099 ceph-mon[24594]: osdmap e60: 8 total, 8 up, 8 in 2024-09-17T00:16:10.663 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:16:10 smithi002 ceph-mon[28203]: mgrmap e17: y(active, since 4m), standbys: x 2024-09-17T00:16:10.664 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:16:10 smithi002 ceph-mon[28203]: Active manager daemon y restarted 2024-09-17T00:16:10.664 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:16:10 smithi002 ceph-mon[28203]: Activating manager daemon y 2024-09-17T00:16:10.664 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:16:10 smithi002 ceph-mon[28203]: osdmap e60: 8 total, 8 up, 8 in 2024-09-17T00:16:10.665 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:16:10 smithi002 ceph-mon[23737]: mgrmap e17: y(active, since 4m), standbys: x 2024-09-17T00:16:10.665 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:16:10 smithi002 ceph-mon[23737]: Active manager daemon y restarted 2024-09-17T00:16:10.665 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:16:10 smithi002 ceph-mon[23737]: Activating manager daemon y 2024-09-17T00:16:10.665 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:16:10 smithi002 ceph-mon[23737]: osdmap e60: 8 total, 8 up, 8 in 2024-09-17T00:16:10.666 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:16:10 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: [17/Sep/2024:00:16:10] ENGINE Bus STARTING 2024-09-17T00:16:10.666 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:16:10 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: CherryPy Checker: 2024-09-17T00:16:10.666 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:16:10 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: The Application mounted at '' has an empty config. 2024-09-17T00:16:10.666 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:16:10 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: 2024-09-17T00:16:10.666 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:16:10 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: [17/Sep/2024:00:16:10] ENGINE Serving on http://:::9283 2024-09-17T00:16:10.667 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:16:10 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: [17/Sep/2024:00:16:10] ENGINE Bus STARTED 2024-09-17T00:16:11.229 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:16:10 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: [17/Sep/2024:00:16:10] ENGINE Bus STARTING 2024-09-17T00:16:11.229 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:16:11 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: [17/Sep/2024:00:16:11] ENGINE Serving on https://172.21.15.2:7150 2024-09-17T00:16:11.229 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:16:11 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: [17/Sep/2024:00:16:11] ENGINE Bus STARTED 2024-09-17T00:16:11.359 INFO:teuthology.orchestra.run.smithi099.stdout:Scheduled node-exporter update... 2024-09-17T00:16:11.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:16:11 smithi099 ceph-mon[24594]: mgrmap e18: y(active, starting, since 0.752987s), standbys: x 2024-09-17T00:16:11.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:16:11 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T00:16:11.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:16:11 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T00:16:11.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:16:11 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T00:16:11.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:16:11 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-17T00:16:11.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:16:11 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-17T00:16:11.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:16:11 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T00:16:11.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:16:11 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T00:16:11.468 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:16:11 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T00:16:11.468 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:16:11 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T00:16:11.468 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:16:11 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T00:16:11.468 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:16:11 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T00:16:11.468 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:16:11 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T00:16:11.468 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:16:11 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T00:16:11.468 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:16:11 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-17T00:16:11.468 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:16:11 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-17T00:16:11.468 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:16:11 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-17T00:16:11.468 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:16:11 smithi099 ceph-mon[24594]: Manager daemon y is now available 2024-09-17T00:16:11.468 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:16:11 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T00:16:11.468 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:16:11 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T00:16:11.468 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:16:11 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:16:11.469 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:16:11 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:16:11.469 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:16:11 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:16:11.469 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:16:11 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T00:16:11.469 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:16:11 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T00:16:11.469 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:16:11 smithi099 ceph-mon[24594]: [17/Sep/2024:00:16:10] ENGINE Bus STARTING 2024-09-17T00:16:11.469 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:16:11 smithi099 ceph-mon[24594]: [17/Sep/2024:00:16:11] ENGINE Serving on https://172.21.15.2:7150 2024-09-17T00:16:11.469 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:16:11 smithi099 ceph-mon[24594]: [17/Sep/2024:00:16:11] ENGINE Bus STARTED 2024-09-17T00:16:11.469 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:16:11 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:16:11.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:16:11 smithi002 ceph-mon[23737]: mgrmap e18: y(active, starting, since 0.752987s), standbys: x 2024-09-17T00:16:11.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:16:11 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T00:16:11.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:16:11 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T00:16:11.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:16:11 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T00:16:11.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:16:11 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-17T00:16:11.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:16:11 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-17T00:16:11.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:16:11 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T00:16:11.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:16:11 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T00:16:11.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:16:11 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T00:16:11.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:16:11 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T00:16:11.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:16:11 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T00:16:11.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:16:11 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T00:16:11.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:16:11 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T00:16:11.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:16:11 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T00:16:11.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:16:11 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-17T00:16:11.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:16:11 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-17T00:16:11.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:16:11 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-17T00:16:11.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:16:11 smithi002 ceph-mon[23737]: Manager daemon y is now available 2024-09-17T00:16:11.731 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:16:11 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T00:16:11.731 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:16:11 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T00:16:11.731 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:16:11 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:16:11.731 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:16:11 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:16:11.731 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:16:11 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:16:11.731 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:16:11 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T00:16:11.731 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:16:11 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T00:16:11.731 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:16:11 smithi002 ceph-mon[23737]: [17/Sep/2024:00:16:10] ENGINE Bus STARTING 2024-09-17T00:16:11.731 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:16:11 smithi002 ceph-mon[23737]: [17/Sep/2024:00:16:11] ENGINE Serving on https://172.21.15.2:7150 2024-09-17T00:16:11.731 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:16:11 smithi002 ceph-mon[23737]: [17/Sep/2024:00:16:11] ENGINE Bus STARTED 2024-09-17T00:16:11.731 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:16:11 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:16:11.731 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:16:11 smithi002 ceph-mon[28203]: mgrmap e18: y(active, starting, since 0.752987s), standbys: x 2024-09-17T00:16:11.732 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:16:11 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T00:16:11.732 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:16:11 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T00:16:11.732 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:16:11 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T00:16:11.732 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:16:11 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-17T00:16:11.732 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:16:11 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-17T00:16:11.732 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:16:11 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T00:16:11.732 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:16:11 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T00:16:11.732 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:16:11 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T00:16:11.732 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:16:11 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T00:16:11.732 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:16:11 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T00:16:11.732 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:16:11 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T00:16:11.732 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:16:11 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T00:16:11.733 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:16:11 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T00:16:11.733 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:16:11 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-17T00:16:11.733 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:16:11 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-17T00:16:11.733 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:16:11 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-17T00:16:11.733 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:16:11 smithi002 ceph-mon[28203]: Manager daemon y is now available 2024-09-17T00:16:11.733 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:16:11 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T00:16:11.733 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:16:11 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T00:16:11.733 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:16:11 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:16:11.733 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:16:11 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:16:11.733 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:16:11 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:16:11.733 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:16:11 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T00:16:11.733 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:16:11 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T00:16:11.733 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:16:11 smithi002 ceph-mon[28203]: [17/Sep/2024:00:16:10] ENGINE Bus STARTING 2024-09-17T00:16:11.733 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:16:11 smithi002 ceph-mon[28203]: [17/Sep/2024:00:16:11] ENGINE Serving on https://172.21.15.2:7150 2024-09-17T00:16:11.734 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:16:11 smithi002 ceph-mon[28203]: [17/Sep/2024:00:16:11] ENGINE Bus STARTED 2024-09-17T00:16:11.734 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:16:11 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:16:12.235 DEBUG:teuthology.orchestra.run.smithi002:node-exporter.a> sudo journalctl -f -n 0 -u ceph-3aad27a2-7489-11ef-bceb-c7b262605968@node-exporter.a.service 2024-09-17T00:16:12.237 DEBUG:teuthology.orchestra.run.smithi099:node-exporter.b> sudo journalctl -f -n 0 -u ceph-3aad27a2-7489-11ef-bceb-c7b262605968@node-exporter.b.service 2024-09-17T00:16:12.240 INFO:tasks.cephadm:Adding alertmanager.a on smithi002 2024-09-17T00:16:12.240 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3aad27a2-7489-11ef-bceb-c7b262605968 -- ceph orch apply alertmanager '1;smithi002=a' 2024-09-17T00:16:12.397 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:16:12 smithi002 ceph-mon[28203]: mgrmap e19: y(active, since 1.76722s), standbys: x 2024-09-17T00:16:12.397 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:16:12 smithi002 ceph-mon[28203]: from='client.14448 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi002=a;smithi099=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:16:12.397 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:16:12 smithi002 ceph-mon[28203]: Saving service node-exporter spec with placement smithi002=a;smithi099=b;count:2 2024-09-17T00:16:12.397 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:16:12 smithi002 ceph-mon[28203]: pgmap v3: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:16:12.398 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:16:12 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:16:12.398 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:16:12 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:16:12.398 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:16:12 smithi002 ceph-mon[23737]: mgrmap e19: y(active, since 1.76722s), standbys: x 2024-09-17T00:16:12.398 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:16:12 smithi002 ceph-mon[23737]: from='client.14448 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi002=a;smithi099=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:16:12.398 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:16:12 smithi002 ceph-mon[23737]: Saving service node-exporter spec with placement smithi002=a;smithi099=b;count:2 2024-09-17T00:16:12.398 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:16:12 smithi002 ceph-mon[23737]: pgmap v3: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:16:12.398 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:16:12 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:16:12.399 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:16:12 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:16:12.466 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:16:12 smithi099 ceph-mon[24594]: mgrmap e19: y(active, since 1.76722s), standbys: x 2024-09-17T00:16:12.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:16:12 smithi099 ceph-mon[24594]: from='client.14448 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi002=a;smithi099=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:16:12.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:16:12 smithi099 ceph-mon[24594]: Saving service node-exporter spec with placement smithi002=a;smithi099=b;count:2 2024-09-17T00:16:12.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:16:12 smithi099 ceph-mon[24594]: pgmap v3: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:16:12.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:16:12 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:16:12.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:16:12 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:16:13.395 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:16:13 smithi099 ceph-mon[24594]: pgmap v4: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:16:13.395 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:16:13 smithi099 ceph-mon[24594]: mgrmap e20: y(active, since 2s), standbys: x 2024-09-17T00:16:13.395 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:16:13 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:16:13.395 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:16:13 smithi099 ceph-mon[24594]: Updating smithi002:/etc/ceph/ceph.conf 2024-09-17T00:16:13.395 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:16:13 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:16:13.396 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:16:13 smithi099 ceph-mon[24594]: Updating smithi002:/etc/ceph/ceph.client.admin.keyring 2024-09-17T00:16:13.396 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:16:13 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:16:13.396 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:16:13 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:16:13.396 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:16:13 smithi099 ceph-mon[24594]: Updating smithi099:/etc/ceph/ceph.conf 2024-09-17T00:16:13.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:16:13 smithi002 ceph-mon[28203]: pgmap v4: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:16:13.776 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:16:13 smithi002 ceph-mon[28203]: mgrmap e20: y(active, since 2s), standbys: x 2024-09-17T00:16:13.776 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:16:13 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:16:13.776 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:16:13 smithi002 ceph-mon[28203]: Updating smithi002:/etc/ceph/ceph.conf 2024-09-17T00:16:13.776 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:16:13 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:16:13.776 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:16:13 smithi002 ceph-mon[28203]: Updating smithi002:/etc/ceph/ceph.client.admin.keyring 2024-09-17T00:16:13.776 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:16:13 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:16:13.776 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:16:13 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:16:13.776 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:16:13 smithi002 ceph-mon[28203]: Updating smithi099:/etc/ceph/ceph.conf 2024-09-17T00:16:13.777 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:16:13 smithi002 ceph-mon[23737]: pgmap v4: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:16:13.777 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:16:13 smithi002 ceph-mon[23737]: mgrmap e20: y(active, since 2s), standbys: x 2024-09-17T00:16:13.777 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:16:13 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:16:13.777 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:16:13 smithi002 ceph-mon[23737]: Updating smithi002:/etc/ceph/ceph.conf 2024-09-17T00:16:13.777 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:16:13 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:16:13.777 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:16:13 smithi002 ceph-mon[23737]: Updating smithi002:/etc/ceph/ceph.client.admin.keyring 2024-09-17T00:16:13.777 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:16:13 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:16:13.777 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:16:13 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:16:13.777 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:16:13 smithi002 ceph-mon[23737]: Updating smithi099:/etc/ceph/ceph.conf 2024-09-17T00:16:14.451 INFO:teuthology.orchestra.run.smithi099.stdout:Scheduled alertmanager update... 2024-09-17T00:16:14.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:16:14 smithi099 ceph-mon[24594]: Updating smithi099:/etc/ceph/ceph.client.admin.keyring 2024-09-17T00:16:14.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:16:14 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:16:14.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:16:14 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:16:14.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:16:14 smithi099 ceph-mon[24594]: Deploying daemon prometheus.a on smithi099 2024-09-17T00:16:14.718 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:16:14 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:16:14.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:16:14 smithi002 ceph-mon[23737]: Updating smithi099:/etc/ceph/ceph.client.admin.keyring 2024-09-17T00:16:14.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:16:14 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:16:14.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:16:14 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:16:14.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:16:14 smithi002 ceph-mon[23737]: Deploying daemon prometheus.a on smithi099 2024-09-17T00:16:14.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:16:14 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:16:14.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:16:14 smithi002 ceph-mon[28203]: Updating smithi099:/etc/ceph/ceph.client.admin.keyring 2024-09-17T00:16:14.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:16:14 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:16:14.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:16:14 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:16:14.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:16:14 smithi002 ceph-mon[28203]: Deploying daemon prometheus.a on smithi099 2024-09-17T00:16:14.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:16:14 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:16:15.094 DEBUG:teuthology.orchestra.run.smithi002:alertmanager.a> sudo journalctl -f -n 0 -u ceph-3aad27a2-7489-11ef-bceb-c7b262605968@alertmanager.a.service 2024-09-17T00:16:15.097 INFO:tasks.cephadm:Adding grafana.a on smithi099 2024-09-17T00:16:15.097 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3aad27a2-7489-11ef-bceb-c7b262605968 -- ceph orch apply grafana '1;smithi099=a' 2024-09-17T00:16:15.716 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:16:15 smithi099 ceph-mon[24594]: pgmap v5: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:16:15.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:16:15 smithi099 ceph-mon[24594]: from='client.24398 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi002=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:16:15.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:16:15 smithi099 ceph-mon[24594]: Saving service alertmanager spec with placement smithi002=a;count:1 2024-09-17T00:16:15.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:16:15 smithi002 ceph-mon[23737]: pgmap v5: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:16:15.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:16:15 smithi002 ceph-mon[23737]: from='client.24398 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi002=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:16:15.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:16:15 smithi002 ceph-mon[23737]: Saving service alertmanager spec with placement smithi002=a;count:1 2024-09-17T00:16:15.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:16:15 smithi002 ceph-mon[28203]: pgmap v5: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:16:15.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:16:15 smithi002 ceph-mon[28203]: from='client.24398 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi002=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:16:15.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:16:15 smithi002 ceph-mon[28203]: Saving service alertmanager spec with placement smithi002=a;count:1 2024-09-17T00:16:17.792 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:16:17 smithi099 ceph-mon[24594]: pgmap v6: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:16:17.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:16:17 smithi002 ceph-mon[23737]: pgmap v6: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:16:17.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:16:17 smithi002 ceph-mon[28203]: pgmap v6: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:16:18.327 INFO:teuthology.orchestra.run.smithi099.stdout:Scheduled grafana update... 2024-09-17T00:16:19.463 DEBUG:teuthology.orchestra.run.smithi099:grafana.a> sudo journalctl -f -n 0 -u ceph-3aad27a2-7489-11ef-bceb-c7b262605968@grafana.a.service 2024-09-17T00:16:19.465 INFO:tasks.cephadm:Setting up client nodes... 2024-09-17T00:16:19.466 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3aad27a2-7489-11ef-bceb-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-09-17T00:16:19.716 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:16:19 smithi099 ceph-mon[24594]: from='client.24404 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi099=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:16:19.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:16:19 smithi099 ceph-mon[24594]: Saving service grafana spec with placement smithi099=a;count:1 2024-09-17T00:16:19.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:16:19 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:16:19.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:16:19 smithi099 ceph-mon[24594]: pgmap v7: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:16:19.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:16:19 smithi002 ceph-mon[23737]: from='client.24404 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi099=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:16:19.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:16:19 smithi002 ceph-mon[23737]: Saving service grafana spec with placement smithi099=a;count:1 2024-09-17T00:16:19.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:16:19 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:16:19.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:16:19 smithi002 ceph-mon[23737]: pgmap v7: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:16:19.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:16:19 smithi002 ceph-mon[28203]: from='client.24404 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi099=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:16:19.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:16:19 smithi002 ceph-mon[28203]: Saving service grafana spec with placement smithi099=a;count:1 2024-09-17T00:16:19.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:16:19 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:16:19.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:16:19 smithi002 ceph-mon[28203]: pgmap v7: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:16:21.397 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:16:21 smithi002 ceph-mon[23737]: pgmap v8: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:16:21.546 INFO:teuthology.orchestra.run.smithi002.stdout:[client.0] 2024-09-17T00:16:21.546 INFO:teuthology.orchestra.run.smithi002.stdout: key = AQBVyuhm6JWUIBAA3Yifv3yvDcVd7xcysZMIRQ== 2024-09-17T00:16:21.716 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:16:21 smithi099 ceph-mon[24594]: pgmap v8: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:16:21.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:16:21 smithi002 ceph-mon[28203]: pgmap v8: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:16:22.116 DEBUG:teuthology.orchestra.run.smithi002:> set -ex 2024-09-17T00:16:22.116 DEBUG:teuthology.orchestra.run.smithi002:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-09-17T00:16:22.116 DEBUG:teuthology.orchestra.run.smithi002:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-17T00:16:22.156 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3aad27a2-7489-11ef-bceb-c7b262605968 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-09-17T00:16:22.716 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:16:22 smithi099 ceph-mon[24594]: from='client.? 172.21.15.2:0/3919489419' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-17T00:16:22.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:16:22 smithi099 ceph-mon[24594]: from='client.? 172.21.15.2:0/3919489419' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-17T00:16:22.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:16:22 smithi002 ceph-mon[28203]: from='client.? 172.21.15.2:0/3919489419' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-17T00:16:22.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:16:22 smithi002 ceph-mon[28203]: from='client.? 172.21.15.2:0/3919489419' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-17T00:16:22.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:16:22 smithi002 ceph-mon[23737]: from='client.? 172.21.15.2:0/3919489419' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-17T00:16:22.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:16:22 smithi002 ceph-mon[23737]: from='client.? 172.21.15.2:0/3919489419' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-17T00:16:23.716 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:16:23 smithi099 ceph-mon[24594]: pgmap v9: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:16:23.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:16:23 smithi002 ceph-mon[23737]: pgmap v9: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:16:23.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:16:23 smithi002 ceph-mon[28203]: pgmap v9: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:16:24.467 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:16:24 smithi099 systemd[1]: Starting Ceph prometheus.a for 3aad27a2-7489-11ef-bceb-c7b262605968... 2024-09-17T00:16:25.145 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:16:25 smithi099 podman[39620]: 2024-09-17 00:16:24.932489888 +0000 UTC m=+0.025022948 image pull 514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d quay.io/prometheus/prometheus:v2.33.4 2024-09-17T00:16:25.409 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:16:25 smithi099 podman[39620]: 2024-09-17 00:16:25.294608182 +0000 UTC m=+0.387141227 container create e644918fd5d891ae7fd53859ff4a91611041c601a3076facc6422c54d193a98b (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-17T00:16:25.711 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:16:25 smithi099 ceph-mon[24594]: pgmap v10: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:16:25.866 INFO:teuthology.orchestra.run.smithi099.stdout:[client.1] 2024-09-17T00:16:25.866 INFO:teuthology.orchestra.run.smithi099.stdout: key = AQBZyuhmZaqSMxAAULPlZx9TDhqWBYMLKvL8lA== 2024-09-17T00:16:25.889 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:16:25 smithi002 ceph-mon[28203]: pgmap v10: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:16:25.889 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:16:25 smithi002 ceph-mon[23737]: pgmap v10: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:16:25.967 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:16:25 smithi099 podman[39620]: 2024-09-17 00:16:25.715187063 +0000 UTC m=+0.807720122 container init e644918fd5d891ae7fd53859ff4a91611041c601a3076facc6422c54d193a98b (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-17T00:16:25.967 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:16:25 smithi099 podman[39620]: 2024-09-17 00:16:25.71907694 +0000 UTC m=+0.811609985 container start e644918fd5d891ae7fd53859ff4a91611041c601a3076facc6422c54d193a98b (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-17T00:16:25.967 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:16:25 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[39646]: ts=2024-09-17T00:16:25.742Z caller=main.go:475 level=info msg="No time or size retention was set so using the default time retention" duration=15d 2024-09-17T00:16:25.967 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:16:25 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[39646]: ts=2024-09-17T00:16:25.742Z caller=main.go:512 level=info msg="Starting Prometheus" version="(version=2.33.4, branch=HEAD, revision=83032011a5d3e6102624fe58241a374a7201fee8)" 2024-09-17T00:16:25.967 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:16:25 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[39646]: ts=2024-09-17T00:16:25.742Z caller=main.go:517 level=info build_context="(go=go1.17.7, user=root@d13bf69e7be8, date=20220222-16:51:28)" 2024-09-17T00:16:25.967 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:16:25 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[39646]: ts=2024-09-17T00:16:25.742Z caller=main.go:518 level=info host_details="(Linux 5.14.0-508.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024 x86_64 smithi099 (none))" 2024-09-17T00:16:25.968 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:16:25 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[39646]: ts=2024-09-17T00:16:25.742Z caller=main.go:519 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-09-17T00:16:25.968 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:16:25 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[39646]: ts=2024-09-17T00:16:25.742Z caller=main.go:520 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-09-17T00:16:25.968 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:16:25 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[39646]: ts=2024-09-17T00:16:25.743Z caller=web.go:570 level=info component=web msg="Start listening for connections" address=:9095 2024-09-17T00:16:25.968 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:16:25 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[39646]: ts=2024-09-17T00:16:25.744Z caller=main.go:923 level=info msg="Starting TSDB ..." 2024-09-17T00:16:25.968 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:16:25 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[39646]: ts=2024-09-17T00:16:25.745Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false 2024-09-17T00:16:25.968 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:16:25 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[39646]: ts=2024-09-17T00:16:25.747Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-09-17T00:16:25.968 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:16:25 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[39646]: ts=2024-09-17T00:16:25.747Z caller=head.go:527 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.692µs 2024-09-17T00:16:25.968 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:16:25 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[39646]: ts=2024-09-17T00:16:25.747Z caller=head.go:533 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-09-17T00:16:25.968 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:16:25 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[39646]: ts=2024-09-17T00:16:25.747Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0 2024-09-17T00:16:25.968 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:16:25 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[39646]: ts=2024-09-17T00:16:25.747Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=19.51µs wal_replay_duration=130.631µs total_replay_duration=164.613µs 2024-09-17T00:16:25.968 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:16:25 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[39646]: ts=2024-09-17T00:16:25.748Z caller=main.go:944 level=info fs_type=XFS_SUPER_MAGIC 2024-09-17T00:16:25.968 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:16:25 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[39646]: ts=2024-09-17T00:16:25.748Z caller=main.go:947 level=info msg="TSDB started" 2024-09-17T00:16:25.968 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:16:25 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[39646]: ts=2024-09-17T00:16:25.748Z caller=main.go:1128 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-09-17T00:16:25.969 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:16:25 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[39646]: ts=2024-09-17T00:16:25.767Z caller=main.go:1165 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=19.205917ms db_storage=654ns remote_storage=1.349µs web_handler=346ns query_engine=732ns scrape=522.195µs scrape_sd=21.631µs notify=1.182µs notify_sd=2.438µs rules=18.411598ms 2024-09-17T00:16:25.969 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:16:25 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[39646]: ts=2024-09-17T00:16:25.767Z caller=main.go:896 level=info msg="Server is ready to receive web requests." 2024-09-17T00:16:25.969 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:16:25 smithi099 bash[39620]: e644918fd5d891ae7fd53859ff4a91611041c601a3076facc6422c54d193a98b 2024-09-17T00:16:25.969 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:16:25 smithi099 systemd[1]: Started Ceph prometheus.a for 3aad27a2-7489-11ef-bceb-c7b262605968. 2024-09-17T00:16:26.469 DEBUG:teuthology.orchestra.run.smithi099:> set -ex 2024-09-17T00:16:26.469 DEBUG:teuthology.orchestra.run.smithi099:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-09-17T00:16:26.469 DEBUG:teuthology.orchestra.run.smithi099:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-09-17T00:16:26.508 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-17T00:16:26.508 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-09-17T00:16:26.508 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 3aad27a2-7489-11ef-bceb-c7b262605968 -- ceph mgr dump --format=json 2024-09-17T00:16:26.754 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T00:16:26.858 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Sep 17 00:16:26 smithi002 bash[42847]: Trying to pull quay.io/prometheus/node-exporter:v1.3.1... 2024-09-17T00:16:26.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:16:26 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:16:26.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:16:26 smithi099 ceph-mon[24594]: Deploying daemon node-exporter.a on smithi002 2024-09-17T00:16:26.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:16:26 smithi099 ceph-mon[24594]: from='client.? 172.21.15.99:0/3665100385' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-17T00:16:26.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:16:26 smithi099 ceph-mon[24594]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-17T00:16:26.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:16:26 smithi099 ceph-mon[24594]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-17T00:16:27.109 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:16:26 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:16:27.110 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:16:26 smithi002 ceph-mon[28203]: Deploying daemon node-exporter.a on smithi002 2024-09-17T00:16:27.110 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:16:26 smithi002 ceph-mon[28203]: from='client.? 172.21.15.99:0/3665100385' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-17T00:16:27.110 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:16:26 smithi002 ceph-mon[28203]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-17T00:16:27.110 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:16:26 smithi002 ceph-mon[28203]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-17T00:16:27.110 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:16:26 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:16:27.110 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:16:26 smithi002 ceph-mon[23737]: Deploying daemon node-exporter.a on smithi002 2024-09-17T00:16:27.110 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:16:26 smithi002 ceph-mon[23737]: from='client.? 172.21.15.99:0/3665100385' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-17T00:16:27.110 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:16:26 smithi002 ceph-mon[23737]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-17T00:16:27.110 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:16:26 smithi002 ceph-mon[23737]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-17T00:16:27.479 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Sep 17 00:16:27 smithi002 bash[42847]: Getting image source signatures 2024-09-17T00:16:27.479 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Sep 17 00:16:27 smithi002 bash[42847]: Copying blob sha256:aa2a8d90b84cb2a9c422e7005cd166a008ccf22ef5d7d4f07128478585ce35ea 2024-09-17T00:16:27.479 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Sep 17 00:16:27 smithi002 bash[42847]: Copying blob sha256:b45d31ee2d7f9f452678a85b0c837c29e12089f31ee8dbac6c8c24dfa4054a30 2024-09-17T00:16:27.479 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Sep 17 00:16:27 smithi002 bash[42847]: Copying blob sha256:b5db1e299295edf3005515ab7879c1df64a33c185d3a7a23aa4dcaa17d26f7b3 2024-09-17T00:16:27.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:16:27 smithi099 ceph-mon[24594]: pgmap v11: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:16:28.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:16:27 smithi002 ceph-mon[23737]: pgmap v11: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:16:28.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:16:27 smithi002 ceph-mon[28203]: pgmap v11: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:16:28.979 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Sep 17 00:16:28 smithi002 bash[42847]: Copying config sha256:1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 2024-09-17T00:16:29.352 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Sep 17 00:16:29 smithi002 bash[42847]: Writing manifest to image destination 2024-09-17T00:16:29.698 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-17T00:16:29.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:16:29 smithi002 ceph-mon[28203]: pgmap v12: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:16:29.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:16:29 smithi002 ceph-mon[28203]: from='client.? 172.21.15.2:0/1498366728' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-09-17T00:16:29.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:16:29 smithi002 ceph-mon[23737]: pgmap v12: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:16:29.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:16:29 smithi002 ceph-mon[23737]: from='client.? 172.21.15.2:0/1498366728' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-09-17T00:16:29.980 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Sep 17 00:16:29 smithi002 podman[42847]: 2024-09-17 00:16:29.591911745 +0000 UTC m=+2.934794198 image pull 1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 quay.io/prometheus/node-exporter:v1.3.1 2024-09-17T00:16:29.980 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Sep 17 00:16:29 smithi002 podman[42847]: 2024-09-17 00:16:29.715861041 +0000 UTC m=+3.058743494 container create d28913b4af3091e941ac8b2d21e27d43655ae89afc90ae230f8a963cb0d31a11 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-17T00:16:30.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:16:29 smithi099 ceph-mon[24594]: pgmap v12: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:16:30.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:16:29 smithi099 ceph-mon[24594]: from='client.? 172.21.15.2:0/1498366728' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-09-17T00:16:30.512 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":20,"active_gid":24337,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6800","nonce":1824194566},{"type":"v1","addr":"172.21.15.2:6801","nonce":1824194566}]},"active_addr":"172.21.15.2:6801/1824194566","active_change":"2024-09-17T00:16:09.587705+0000","active_mgr_features":4540138303579357183,"available":true,"standbys":[{"gid":24356,"name":"x","mgr_features":4540138303579357183,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.2:8443/","prometheus":"http://172.21.15.2:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":60,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.2:0","nonce":2517426025}]},{"addrvec":[{"type":"v2","addr":"172.21.15.2:0","nonce":3887064165}]},{"addrvec":[{"type":"v2","addr":"172.21.15.2:0","nonce":4027284037}]},{"addrvec":[{"type":"v2","addr":"172.21.15.2:0","nonce":3574422273}]}]}} 2024-09-17T00:16:30.516 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-09-17T00:16:30.516 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-09-17T00:16:30.516 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 3aad27a2-7489-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-17T00:16:30.735 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Sep 17 00:16:30 smithi002 podman[42847]: 2024-09-17 00:16:30.655277403 +0000 UTC m=+3.998159864 container init d28913b4af3091e941ac8b2d21e27d43655ae89afc90ae230f8a963cb0d31a11 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-17T00:16:30.736 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Sep 17 00:16:30 smithi002 podman[42847]: 2024-09-17 00:16:30.659789287 +0000 UTC m=+4.002671749 container start d28913b4af3091e941ac8b2d21e27d43655ae89afc90ae230f8a963cb0d31a11 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-17T00:16:30.736 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Sep 17 00:16:30 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-a[43041]: ts=2024-09-17T00:16:30.664Z caller=node_exporter.go:182 level=info msg="Starting node_exporter" version="(version=1.3.1, branch=HEAD, revision=a2321e7b940ddcff26873612bccdf7cd4c42b6b6)" 2024-09-17T00:16:30.736 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Sep 17 00:16:30 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-a[43041]: ts=2024-09-17T00:16:30.664Z caller=node_exporter.go:183 level=info msg="Build context" build_context="(go=go1.17.3, user=root@243aafa5525c, date=20211205-11:09:49)" 2024-09-17T00:16:30.736 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Sep 17 00:16:30 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-a[43041]: ts=2024-09-17T00:16:30.665Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+)($|/) 2024-09-17T00:16:30.736 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Sep 17 00:16:30 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-a[43041]: ts=2024-09-17T00:16:30.665Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-09-17T00:16:30.736 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Sep 17 00:16:30 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-a[43041]: ts=2024-09-17T00:16:30.665Z caller=node_exporter.go:108 level=info msg="Enabled collectors" 2024-09-17T00:16:30.736 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Sep 17 00:16:30 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-a[43041]: ts=2024-09-17T00:16:30.665Z caller=node_exporter.go:115 level=info collector=arp 2024-09-17T00:16:30.736 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Sep 17 00:16:30 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-a[43041]: ts=2024-09-17T00:16:30.665Z caller=node_exporter.go:115 level=info collector=bcache 2024-09-17T00:16:30.736 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Sep 17 00:16:30 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-a[43041]: ts=2024-09-17T00:16:30.665Z caller=node_exporter.go:115 level=info collector=bonding 2024-09-17T00:16:30.736 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Sep 17 00:16:30 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-a[43041]: ts=2024-09-17T00:16:30.665Z caller=node_exporter.go:115 level=info collector=btrfs 2024-09-17T00:16:30.736 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Sep 17 00:16:30 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-a[43041]: ts=2024-09-17T00:16:30.665Z caller=node_exporter.go:115 level=info collector=conntrack 2024-09-17T00:16:30.736 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Sep 17 00:16:30 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-a[43041]: ts=2024-09-17T00:16:30.665Z caller=node_exporter.go:115 level=info collector=cpu 2024-09-17T00:16:30.737 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Sep 17 00:16:30 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-a[43041]: ts=2024-09-17T00:16:30.666Z caller=node_exporter.go:115 level=info collector=cpufreq 2024-09-17T00:16:30.737 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Sep 17 00:16:30 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-a[43041]: ts=2024-09-17T00:16:30.666Z caller=node_exporter.go:115 level=info collector=diskstats 2024-09-17T00:16:30.737 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Sep 17 00:16:30 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-a[43041]: ts=2024-09-17T00:16:30.666Z caller=node_exporter.go:115 level=info collector=dmi 2024-09-17T00:16:30.737 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Sep 17 00:16:30 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-a[43041]: ts=2024-09-17T00:16:30.666Z caller=node_exporter.go:115 level=info collector=edac 2024-09-17T00:16:30.737 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Sep 17 00:16:30 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-a[43041]: ts=2024-09-17T00:16:30.666Z caller=node_exporter.go:115 level=info collector=entropy 2024-09-17T00:16:30.737 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Sep 17 00:16:30 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-a[43041]: ts=2024-09-17T00:16:30.666Z caller=node_exporter.go:115 level=info collector=fibrechannel 2024-09-17T00:16:30.737 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Sep 17 00:16:30 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-a[43041]: ts=2024-09-17T00:16:30.666Z caller=node_exporter.go:115 level=info collector=filefd 2024-09-17T00:16:30.737 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Sep 17 00:16:30 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-a[43041]: ts=2024-09-17T00:16:30.666Z caller=node_exporter.go:115 level=info collector=filesystem 2024-09-17T00:16:30.737 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Sep 17 00:16:30 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-a[43041]: ts=2024-09-17T00:16:30.666Z caller=node_exporter.go:115 level=info collector=hwmon 2024-09-17T00:16:30.737 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Sep 17 00:16:30 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-a[43041]: ts=2024-09-17T00:16:30.666Z caller=node_exporter.go:115 level=info collector=infiniband 2024-09-17T00:16:30.737 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Sep 17 00:16:30 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-a[43041]: ts=2024-09-17T00:16:30.666Z caller=node_exporter.go:115 level=info collector=ipvs 2024-09-17T00:16:30.737 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Sep 17 00:16:30 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-a[43041]: ts=2024-09-17T00:16:30.666Z caller=node_exporter.go:115 level=info collector=loadavg 2024-09-17T00:16:30.737 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Sep 17 00:16:30 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-a[43041]: ts=2024-09-17T00:16:30.666Z caller=node_exporter.go:115 level=info collector=mdadm 2024-09-17T00:16:30.737 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Sep 17 00:16:30 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-a[43041]: ts=2024-09-17T00:16:30.666Z caller=node_exporter.go:115 level=info collector=meminfo 2024-09-17T00:16:30.738 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Sep 17 00:16:30 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-a[43041]: ts=2024-09-17T00:16:30.666Z caller=node_exporter.go:115 level=info collector=netclass 2024-09-17T00:16:30.738 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Sep 17 00:16:30 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-a[43041]: ts=2024-09-17T00:16:30.666Z caller=node_exporter.go:115 level=info collector=netdev 2024-09-17T00:16:30.738 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Sep 17 00:16:30 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-a[43041]: ts=2024-09-17T00:16:30.666Z caller=node_exporter.go:115 level=info collector=netstat 2024-09-17T00:16:30.738 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Sep 17 00:16:30 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-a[43041]: ts=2024-09-17T00:16:30.666Z caller=node_exporter.go:115 level=info collector=nfs 2024-09-17T00:16:30.738 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Sep 17 00:16:30 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-a[43041]: ts=2024-09-17T00:16:30.666Z caller=node_exporter.go:115 level=info collector=nfsd 2024-09-17T00:16:30.738 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Sep 17 00:16:30 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-a[43041]: ts=2024-09-17T00:16:30.666Z caller=node_exporter.go:115 level=info collector=nvme 2024-09-17T00:16:30.738 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Sep 17 00:16:30 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-a[43041]: ts=2024-09-17T00:16:30.666Z caller=node_exporter.go:115 level=info collector=os 2024-09-17T00:16:30.738 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Sep 17 00:16:30 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-a[43041]: ts=2024-09-17T00:16:30.666Z caller=node_exporter.go:115 level=info collector=powersupplyclass 2024-09-17T00:16:30.738 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Sep 17 00:16:30 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-a[43041]: ts=2024-09-17T00:16:30.666Z caller=node_exporter.go:115 level=info collector=pressure 2024-09-17T00:16:30.738 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Sep 17 00:16:30 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-a[43041]: ts=2024-09-17T00:16:30.666Z caller=node_exporter.go:115 level=info collector=rapl 2024-09-17T00:16:30.738 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Sep 17 00:16:30 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-a[43041]: ts=2024-09-17T00:16:30.666Z caller=node_exporter.go:115 level=info collector=schedstat 2024-09-17T00:16:30.738 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Sep 17 00:16:30 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-a[43041]: ts=2024-09-17T00:16:30.666Z caller=node_exporter.go:115 level=info collector=sockstat 2024-09-17T00:16:30.738 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Sep 17 00:16:30 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-a[43041]: ts=2024-09-17T00:16:30.666Z caller=node_exporter.go:115 level=info collector=softnet 2024-09-17T00:16:30.739 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Sep 17 00:16:30 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-a[43041]: ts=2024-09-17T00:16:30.666Z caller=node_exporter.go:115 level=info collector=stat 2024-09-17T00:16:30.739 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Sep 17 00:16:30 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-a[43041]: ts=2024-09-17T00:16:30.666Z caller=node_exporter.go:115 level=info collector=tapestats 2024-09-17T00:16:30.739 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Sep 17 00:16:30 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-a[43041]: ts=2024-09-17T00:16:30.666Z caller=node_exporter.go:115 level=info collector=textfile 2024-09-17T00:16:30.739 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Sep 17 00:16:30 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-a[43041]: ts=2024-09-17T00:16:30.666Z caller=node_exporter.go:115 level=info collector=thermal_zone 2024-09-17T00:16:30.739 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Sep 17 00:16:30 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-a[43041]: ts=2024-09-17T00:16:30.666Z caller=node_exporter.go:115 level=info collector=time 2024-09-17T00:16:30.739 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Sep 17 00:16:30 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-a[43041]: ts=2024-09-17T00:16:30.666Z caller=node_exporter.go:115 level=info collector=udp_queues 2024-09-17T00:16:30.739 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Sep 17 00:16:30 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-a[43041]: ts=2024-09-17T00:16:30.666Z caller=node_exporter.go:115 level=info collector=uname 2024-09-17T00:16:30.739 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Sep 17 00:16:30 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-a[43041]: ts=2024-09-17T00:16:30.666Z caller=node_exporter.go:115 level=info collector=vmstat 2024-09-17T00:16:30.739 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Sep 17 00:16:30 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-a[43041]: ts=2024-09-17T00:16:30.666Z caller=node_exporter.go:115 level=info collector=xfs 2024-09-17T00:16:30.739 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Sep 17 00:16:30 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-a[43041]: ts=2024-09-17T00:16:30.666Z caller=node_exporter.go:115 level=info collector=zfs 2024-09-17T00:16:30.739 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Sep 17 00:16:30 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-a[43041]: ts=2024-09-17T00:16:30.666Z caller=node_exporter.go:199 level=info msg="Listening on" address=:9100 2024-09-17T00:16:30.740 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Sep 17 00:16:30 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-a[43041]: ts=2024-09-17T00:16:30.666Z caller=tls_config.go:195 level=info msg="TLS is disabled." http2=false 2024-09-17T00:16:30.740 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Sep 17 00:16:30 smithi002 bash[42847]: d28913b4af3091e941ac8b2d21e27d43655ae89afc90ae230f8a963cb0d31a11 2024-09-17T00:16:30.778 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T00:16:31.036 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Sep 17 00:16:30 smithi002 systemd[1]: Started Ceph node-exporter.a for 3aad27a2-7489-11ef-bceb-c7b262605968. 2024-09-17T00:16:31.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:16:31 smithi099 ceph-mon[24594]: pgmap v13: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:16:31.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:16:31 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:16:31.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:16:31 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:16:31.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:16:31 smithi099 ceph-mon[24594]: Deploying daemon node-exporter.b on smithi099 2024-09-17T00:16:31.717 INFO:journalctl@ceph.node-exporter.b.smithi099.stdout:Sep 17 00:16:31 smithi099 bash[39933]: Trying to pull quay.io/prometheus/node-exporter:v1.3.1... 2024-09-17T00:16:31.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:16:31 smithi002 ceph-mon[23737]: pgmap v13: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:16:31.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:16:31 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:16:31.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:16:31 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:16:31.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:16:31 smithi002 ceph-mon[23737]: Deploying daemon node-exporter.b on smithi099 2024-09-17T00:16:31.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:16:31 smithi002 ceph-mon[28203]: pgmap v13: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:16:31.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:16:31 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:16:31.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:16:31 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:16:31.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:16:31 smithi002 ceph-mon[28203]: Deploying daemon node-exporter.b on smithi099 2024-09-17T00:16:32.467 INFO:journalctl@ceph.node-exporter.b.smithi099.stdout:Sep 17 00:16:31 smithi099 bash[39933]: Getting image source signatures 2024-09-17T00:16:32.467 INFO:journalctl@ceph.node-exporter.b.smithi099.stdout:Sep 17 00:16:31 smithi099 bash[39933]: Copying blob sha256:aa2a8d90b84cb2a9c422e7005cd166a008ccf22ef5d7d4f07128478585ce35ea 2024-09-17T00:16:32.467 INFO:journalctl@ceph.node-exporter.b.smithi099.stdout:Sep 17 00:16:31 smithi099 bash[39933]: Copying blob sha256:b45d31ee2d7f9f452678a85b0c837c29e12089f31ee8dbac6c8c24dfa4054a30 2024-09-17T00:16:32.467 INFO:journalctl@ceph.node-exporter.b.smithi099.stdout:Sep 17 00:16:31 smithi099 bash[39933]: Copying blob sha256:b5db1e299295edf3005515ab7879c1df64a33c185d3a7a23aa4dcaa17d26f7b3 2024-09-17T00:16:32.605 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-17T00:16:32.605 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":60,"fsid":"3aad27a2-7489-11ef-bceb-c7b262605968","created":"2024-09-17T00:10:48.442986+0000","modified":"2024-09-17T00:16:09.587286+0000","last_up_change":"2024-09-17T00:15:50.239941+0000","last_in_change":"2024-09-17T00:15:32.949068+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T00:13:54.650037+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"2030f6f2-3659-4b78-bca5-b74d15218c34","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":2656745758},{"type":"v1","addr":"172.21.15.2:6803","nonce":2656745758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":2656745758},{"type":"v1","addr":"172.21.15.2:6805","nonce":2656745758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":2656745758},{"type":"v1","addr":"172.21.15.2:6809","nonce":2656745758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":2656745758},{"type":"v1","addr":"172.21.15.2:6807","nonce":2656745758}]},"public_addr":"172.21.15.2:6803/2656745758","cluster_addr":"172.21.15.2:6805/2656745758","heartbeat_back_addr":"172.21.15.2:6809/2656745758","heartbeat_front_addr":"172.21.15.2:6807/2656745758","state":["exists","up"]},{"osd":1,"uuid":"e3883c47-fd8d-444f-8057-25d08b59af31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":981024217},{"type":"v1","addr":"172.21.15.2:6811","nonce":981024217}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":981024217},{"type":"v1","addr":"172.21.15.2:6813","nonce":981024217}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":981024217},{"type":"v1","addr":"172.21.15.2:6817","nonce":981024217}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":981024217},{"type":"v1","addr":"172.21.15.2:6815","nonce":981024217}]},"public_addr":"172.21.15.2:6811/981024217","cluster_addr":"172.21.15.2:6813/981024217","heartbeat_back_addr":"172.21.15.2:6817/981024217","heartbeat_front_addr":"172.21.15.2:6815/981024217","state":["exists","up"]},{"osd":2,"uuid":"9d042533-63e5-4f38-a8db-10b61e9b39d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":2219840596},{"type":"v1","addr":"172.21.15.2:6819","nonce":2219840596}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":2219840596},{"type":"v1","addr":"172.21.15.2:6821","nonce":2219840596}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":2219840596},{"type":"v1","addr":"172.21.15.2:6825","nonce":2219840596}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":2219840596},{"type":"v1","addr":"172.21.15.2:6823","nonce":2219840596}]},"public_addr":"172.21.15.2:6819/2219840596","cluster_addr":"172.21.15.2:6821/2219840596","heartbeat_back_addr":"172.21.15.2:6825/2219840596","heartbeat_front_addr":"172.21.15.2:6823/2219840596","state":["exists","up"]},{"osd":3,"uuid":"d9475026-f3ce-4b45-b5de-780c78dbc233","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6826","nonce":1628413766},{"type":"v1","addr":"172.21.15.2:6827","nonce":1628413766}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":1628413766},{"type":"v1","addr":"172.21.15.2:6829","nonce":1628413766}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":1628413766},{"type":"v1","addr":"172.21.15.2:6833","nonce":1628413766}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":1628413766},{"type":"v1","addr":"172.21.15.2:6831","nonce":1628413766}]},"public_addr":"172.21.15.2:6827/1628413766","cluster_addr":"172.21.15.2:6829/1628413766","heartbeat_back_addr":"172.21.15.2:6833/1628413766","heartbeat_front_addr":"172.21.15.2:6831/1628413766","state":["exists","up"]},{"osd":4,"uuid":"f8f0575a-3699-4e8f-b969-e4b6973708df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6800","nonce":2601657006},{"type":"v1","addr":"172.21.15.99:6801","nonce":2601657006}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6802","nonce":2601657006},{"type":"v1","addr":"172.21.15.99:6803","nonce":2601657006}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6806","nonce":2601657006},{"type":"v1","addr":"172.21.15.99:6807","nonce":2601657006}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6804","nonce":2601657006},{"type":"v1","addr":"172.21.15.99:6805","nonce":2601657006}]},"public_addr":"172.21.15.99:6801/2601657006","cluster_addr":"172.21.15.99:6803/2601657006","heartbeat_back_addr":"172.21.15.99:6807/2601657006","heartbeat_front_addr":"172.21.15.99:6805/2601657006","state":["exists","up"]},{"osd":5,"uuid":"c0ba894b-bcbd-40e5-b00d-73dbddc87d59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6808","nonce":1930951601},{"type":"v1","addr":"172.21.15.99:6809","nonce":1930951601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6810","nonce":1930951601},{"type":"v1","addr":"172.21.15.99:6811","nonce":1930951601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6814","nonce":1930951601},{"type":"v1","addr":"172.21.15.99:6815","nonce":1930951601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6812","nonce":1930951601},{"type":"v1","addr":"172.21.15.99:6813","nonce":1930951601}]},"public_addr":"172.21.15.99:6809/1930951601","cluster_addr":"172.21.15.99:6811/1930951601","heartbeat_back_addr":"172.21.15.99:6815/1930951601","heartbeat_front_addr":"172.21.15.99:6813/1930951601","state":["exists","up"]},{"osd":6,"uuid":"9aeb407a-4fd9-415c-aa66-1e66d8aaa700","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6816","nonce":462153986},{"type":"v1","addr":"172.21.15.99:6817","nonce":462153986}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6818","nonce":462153986},{"type":"v1","addr":"172.21.15.99:6819","nonce":462153986}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6822","nonce":462153986},{"type":"v1","addr":"172.21.15.99:6823","nonce":462153986}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6820","nonce":462153986},{"type":"v1","addr":"172.21.15.99:6821","nonce":462153986}]},"public_addr":"172.21.15.99:6817/462153986","cluster_addr":"172.21.15.99:6819/462153986","heartbeat_back_addr":"172.21.15.99:6823/462153986","heartbeat_front_addr":"172.21.15.99:6821/462153986","state":["exists","up"]},{"osd":7,"uuid":"9b87a926-31b4-4937-b425-b52e773f79e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6824","nonce":544554822},{"type":"v1","addr":"172.21.15.99:6825","nonce":544554822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6826","nonce":544554822},{"type":"v1","addr":"172.21.15.99:6827","nonce":544554822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6830","nonce":544554822},{"type":"v1","addr":"172.21.15.99:6831","nonce":544554822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6828","nonce":544554822},{"type":"v1","addr":"172.21.15.99:6829","nonce":544554822}]},"public_addr":"172.21.15.99:6825/544554822","cluster_addr":"172.21.15.99:6827/544554822","heartbeat_back_addr":"172.21.15.99:6831/544554822","heartbeat_front_addr":"172.21.15.99:6829/544554822","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T00:13:02.874193+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T00:13:28.064593+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T00:13:51.522483+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T00:14:15.866757+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T00:14:37.409107+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T00:15:00.424239+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T00:15:24.796694+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T00:15:48.372673+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.2:0/1056079102":"2024-09-18T00:16:09.587221+0000","172.21.15.2:6801/3807888742":"2024-09-18T00:16:09.587221+0000","172.21.15.2:6800/3807888742":"2024-09-18T00:16:09.587221+0000","172.21.15.2:0/1754068649":"2024-09-18T00:16:09.587221+0000","172.21.15.2:0/1780251331":"2024-09-18T00:11:44.094172+0000","172.21.15.2:0/1474587242":"2024-09-18T00:16:09.587221+0000","172.21.15.2:0/824729147":"2024-09-18T00:11:44.094172+0000","172.21.15.2:6801/2522360616":"2024-09-18T00:11:44.094172+0000","172.21.15.2:6800/2522360616":"2024-09-18T00:11:44.094172+0000","172.21.15.2:0/3091500665":"2024-09-18T00:11:44.094172+0000","172.21.15.2:0/3536227112":"2024-09-18T00:11:20.938701+0000","172.21.15.2:6800/3667297605":"2024-09-18T00:11:20.938701+0000","172.21.15.2:0/626695359":"2024-09-18T00:16:09.587221+0000","172.21.15.2:6801/3667297605":"2024-09-18T00:11:20.938701+0000","172.21.15.2:0/3867827864":"2024-09-18T00:11:20.938701+0000","172.21.15.2:0/174032459":"2024-09-18T00:11:20.938701+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T00:16:33.347 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-09-17T00:16:33.348 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 3aad27a2-7489-11ef-bceb-c7b262605968 -- ceph osd dump --format=json 2024-09-17T00:16:33.466 INFO:journalctl@ceph.node-exporter.b.smithi099.stdout:Sep 17 00:16:33 smithi099 bash[39933]: Copying config sha256:1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 2024-09-17T00:16:33.467 INFO:journalctl@ceph.node-exporter.b.smithi099.stdout:Sep 17 00:16:33 smithi099 bash[39933]: Writing manifest to image destination 2024-09-17T00:16:33.567 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T00:16:33.689 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:16:33 smithi002 ceph-mon[28203]: pgmap v14: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:16:33.689 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:16:33 smithi002 ceph-mon[28203]: from='client.? 172.21.15.2:0/318653630' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-17T00:16:33.689 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:16:33 smithi002 ceph-mon[23737]: pgmap v14: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:16:33.689 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:16:33 smithi002 ceph-mon[23737]: from='client.? 172.21.15.2:0/318653630' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-17T00:16:33.827 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:16:33 smithi099 ceph-mon[24594]: pgmap v14: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:16:33.827 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:16:33 smithi099 ceph-mon[24594]: from='client.? 172.21.15.2:0/318653630' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-17T00:16:33.827 INFO:journalctl@ceph.node-exporter.b.smithi099.stdout:Sep 17 00:16:33 smithi099 podman[39933]: 2024-09-17 00:16:33.624052589 +0000 UTC m=+2.076471847 image pull 1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 quay.io/prometheus/node-exporter:v1.3.1 2024-09-17T00:16:33.827 INFO:journalctl@ceph.node-exporter.b.smithi099.stdout:Sep 17 00:16:33 smithi099 podman[39933]: 2024-09-17 00:16:33.706120117 +0000 UTC m=+2.158539377 container create 352ae73fe2a7d92d222a4b36681d326961a29b84383478e8bc975f417e124933 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-17T00:16:34.217 INFO:journalctl@ceph.node-exporter.b.smithi099.stdout:Sep 17 00:16:33 smithi099 podman[39933]: 2024-09-17 00:16:33.92699118 +0000 UTC m=+2.379410442 container init 352ae73fe2a7d92d222a4b36681d326961a29b84383478e8bc975f417e124933 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-17T00:16:34.217 INFO:journalctl@ceph.node-exporter.b.smithi099.stdout:Sep 17 00:16:33 smithi099 podman[39933]: 2024-09-17 00:16:33.930129501 +0000 UTC m=+2.382548758 container start 352ae73fe2a7d92d222a4b36681d326961a29b84383478e8bc975f417e124933 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-17T00:16:34.217 INFO:journalctl@ceph.node-exporter.b.smithi099.stdout:Sep 17 00:16:33 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-b[40010]: ts=2024-09-17T00:16:33.933Z caller=node_exporter.go:182 level=info msg="Starting node_exporter" version="(version=1.3.1, branch=HEAD, revision=a2321e7b940ddcff26873612bccdf7cd4c42b6b6)" 2024-09-17T00:16:34.217 INFO:journalctl@ceph.node-exporter.b.smithi099.stdout:Sep 17 00:16:33 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-b[40010]: ts=2024-09-17T00:16:33.934Z caller=node_exporter.go:183 level=info msg="Build context" build_context="(go=go1.17.3, user=root@243aafa5525c, date=20211205-11:09:49)" 2024-09-17T00:16:34.217 INFO:journalctl@ceph.node-exporter.b.smithi099.stdout:Sep 17 00:16:33 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-b[40010]: ts=2024-09-17T00:16:33.934Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+)($|/) 2024-09-17T00:16:34.218 INFO:journalctl@ceph.node-exporter.b.smithi099.stdout:Sep 17 00:16:33 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-b[40010]: ts=2024-09-17T00:16:33.934Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-09-17T00:16:34.218 INFO:journalctl@ceph.node-exporter.b.smithi099.stdout:Sep 17 00:16:33 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-b[40010]: ts=2024-09-17T00:16:33.935Z caller=node_exporter.go:108 level=info msg="Enabled collectors" 2024-09-17T00:16:34.218 INFO:journalctl@ceph.node-exporter.b.smithi099.stdout:Sep 17 00:16:33 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-b[40010]: ts=2024-09-17T00:16:33.935Z caller=node_exporter.go:115 level=info collector=arp 2024-09-17T00:16:34.218 INFO:journalctl@ceph.node-exporter.b.smithi099.stdout:Sep 17 00:16:33 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-b[40010]: ts=2024-09-17T00:16:33.935Z caller=node_exporter.go:115 level=info collector=bcache 2024-09-17T00:16:34.218 INFO:journalctl@ceph.node-exporter.b.smithi099.stdout:Sep 17 00:16:33 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-b[40010]: ts=2024-09-17T00:16:33.935Z caller=node_exporter.go:115 level=info collector=bonding 2024-09-17T00:16:34.218 INFO:journalctl@ceph.node-exporter.b.smithi099.stdout:Sep 17 00:16:33 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-b[40010]: ts=2024-09-17T00:16:33.935Z caller=node_exporter.go:115 level=info collector=btrfs 2024-09-17T00:16:34.218 INFO:journalctl@ceph.node-exporter.b.smithi099.stdout:Sep 17 00:16:33 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-b[40010]: ts=2024-09-17T00:16:33.935Z caller=node_exporter.go:115 level=info collector=conntrack 2024-09-17T00:16:34.218 INFO:journalctl@ceph.node-exporter.b.smithi099.stdout:Sep 17 00:16:33 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-b[40010]: ts=2024-09-17T00:16:33.935Z caller=node_exporter.go:115 level=info collector=cpu 2024-09-17T00:16:34.218 INFO:journalctl@ceph.node-exporter.b.smithi099.stdout:Sep 17 00:16:33 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-b[40010]: ts=2024-09-17T00:16:33.935Z caller=node_exporter.go:115 level=info collector=cpufreq 2024-09-17T00:16:34.219 INFO:journalctl@ceph.node-exporter.b.smithi099.stdout:Sep 17 00:16:33 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-b[40010]: ts=2024-09-17T00:16:33.935Z caller=node_exporter.go:115 level=info collector=diskstats 2024-09-17T00:16:34.219 INFO:journalctl@ceph.node-exporter.b.smithi099.stdout:Sep 17 00:16:33 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-b[40010]: ts=2024-09-17T00:16:33.935Z caller=node_exporter.go:115 level=info collector=dmi 2024-09-17T00:16:34.219 INFO:journalctl@ceph.node-exporter.b.smithi099.stdout:Sep 17 00:16:33 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-b[40010]: ts=2024-09-17T00:16:33.935Z caller=node_exporter.go:115 level=info collector=edac 2024-09-17T00:16:34.219 INFO:journalctl@ceph.node-exporter.b.smithi099.stdout:Sep 17 00:16:33 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-b[40010]: ts=2024-09-17T00:16:33.935Z caller=node_exporter.go:115 level=info collector=entropy 2024-09-17T00:16:34.219 INFO:journalctl@ceph.node-exporter.b.smithi099.stdout:Sep 17 00:16:33 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-b[40010]: ts=2024-09-17T00:16:33.935Z caller=node_exporter.go:115 level=info collector=fibrechannel 2024-09-17T00:16:34.219 INFO:journalctl@ceph.node-exporter.b.smithi099.stdout:Sep 17 00:16:33 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-b[40010]: ts=2024-09-17T00:16:33.935Z caller=node_exporter.go:115 level=info collector=filefd 2024-09-17T00:16:34.219 INFO:journalctl@ceph.node-exporter.b.smithi099.stdout:Sep 17 00:16:33 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-b[40010]: ts=2024-09-17T00:16:33.935Z caller=node_exporter.go:115 level=info collector=filesystem 2024-09-17T00:16:34.219 INFO:journalctl@ceph.node-exporter.b.smithi099.stdout:Sep 17 00:16:33 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-b[40010]: ts=2024-09-17T00:16:33.935Z caller=node_exporter.go:115 level=info collector=hwmon 2024-09-17T00:16:34.219 INFO:journalctl@ceph.node-exporter.b.smithi099.stdout:Sep 17 00:16:33 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-b[40010]: ts=2024-09-17T00:16:33.935Z caller=node_exporter.go:115 level=info collector=infiniband 2024-09-17T00:16:34.219 INFO:journalctl@ceph.node-exporter.b.smithi099.stdout:Sep 17 00:16:33 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-b[40010]: ts=2024-09-17T00:16:33.935Z caller=node_exporter.go:115 level=info collector=ipvs 2024-09-17T00:16:34.219 INFO:journalctl@ceph.node-exporter.b.smithi099.stdout:Sep 17 00:16:33 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-b[40010]: ts=2024-09-17T00:16:33.935Z caller=node_exporter.go:115 level=info collector=loadavg 2024-09-17T00:16:34.219 INFO:journalctl@ceph.node-exporter.b.smithi099.stdout:Sep 17 00:16:33 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-b[40010]: ts=2024-09-17T00:16:33.935Z caller=node_exporter.go:115 level=info collector=mdadm 2024-09-17T00:16:34.219 INFO:journalctl@ceph.node-exporter.b.smithi099.stdout:Sep 17 00:16:33 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-b[40010]: ts=2024-09-17T00:16:33.935Z caller=node_exporter.go:115 level=info collector=meminfo 2024-09-17T00:16:34.219 INFO:journalctl@ceph.node-exporter.b.smithi099.stdout:Sep 17 00:16:33 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-b[40010]: ts=2024-09-17T00:16:33.935Z caller=node_exporter.go:115 level=info collector=netclass 2024-09-17T00:16:34.220 INFO:journalctl@ceph.node-exporter.b.smithi099.stdout:Sep 17 00:16:33 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-b[40010]: ts=2024-09-17T00:16:33.935Z caller=node_exporter.go:115 level=info collector=netdev 2024-09-17T00:16:34.220 INFO:journalctl@ceph.node-exporter.b.smithi099.stdout:Sep 17 00:16:33 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-b[40010]: ts=2024-09-17T00:16:33.935Z caller=node_exporter.go:115 level=info collector=netstat 2024-09-17T00:16:34.220 INFO:journalctl@ceph.node-exporter.b.smithi099.stdout:Sep 17 00:16:33 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-b[40010]: ts=2024-09-17T00:16:33.935Z caller=node_exporter.go:115 level=info collector=nfs 2024-09-17T00:16:34.220 INFO:journalctl@ceph.node-exporter.b.smithi099.stdout:Sep 17 00:16:33 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-b[40010]: ts=2024-09-17T00:16:33.935Z caller=node_exporter.go:115 level=info collector=nfsd 2024-09-17T00:16:34.220 INFO:journalctl@ceph.node-exporter.b.smithi099.stdout:Sep 17 00:16:33 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-b[40010]: ts=2024-09-17T00:16:33.935Z caller=node_exporter.go:115 level=info collector=nvme 2024-09-17T00:16:34.220 INFO:journalctl@ceph.node-exporter.b.smithi099.stdout:Sep 17 00:16:33 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-b[40010]: ts=2024-09-17T00:16:33.935Z caller=node_exporter.go:115 level=info collector=os 2024-09-17T00:16:34.220 INFO:journalctl@ceph.node-exporter.b.smithi099.stdout:Sep 17 00:16:33 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-b[40010]: ts=2024-09-17T00:16:33.935Z caller=node_exporter.go:115 level=info collector=powersupplyclass 2024-09-17T00:16:34.220 INFO:journalctl@ceph.node-exporter.b.smithi099.stdout:Sep 17 00:16:33 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-b[40010]: ts=2024-09-17T00:16:33.935Z caller=node_exporter.go:115 level=info collector=pressure 2024-09-17T00:16:34.220 INFO:journalctl@ceph.node-exporter.b.smithi099.stdout:Sep 17 00:16:33 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-b[40010]: ts=2024-09-17T00:16:33.935Z caller=node_exporter.go:115 level=info collector=rapl 2024-09-17T00:16:34.220 INFO:journalctl@ceph.node-exporter.b.smithi099.stdout:Sep 17 00:16:33 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-b[40010]: ts=2024-09-17T00:16:33.935Z caller=node_exporter.go:115 level=info collector=schedstat 2024-09-17T00:16:34.220 INFO:journalctl@ceph.node-exporter.b.smithi099.stdout:Sep 17 00:16:33 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-b[40010]: ts=2024-09-17T00:16:33.935Z caller=node_exporter.go:115 level=info collector=sockstat 2024-09-17T00:16:34.220 INFO:journalctl@ceph.node-exporter.b.smithi099.stdout:Sep 17 00:16:33 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-b[40010]: ts=2024-09-17T00:16:33.935Z caller=node_exporter.go:115 level=info collector=softnet 2024-09-17T00:16:34.220 INFO:journalctl@ceph.node-exporter.b.smithi099.stdout:Sep 17 00:16:33 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-b[40010]: ts=2024-09-17T00:16:33.935Z caller=node_exporter.go:115 level=info collector=stat 2024-09-17T00:16:34.221 INFO:journalctl@ceph.node-exporter.b.smithi099.stdout:Sep 17 00:16:33 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-b[40010]: ts=2024-09-17T00:16:33.935Z caller=node_exporter.go:115 level=info collector=tapestats 2024-09-17T00:16:34.221 INFO:journalctl@ceph.node-exporter.b.smithi099.stdout:Sep 17 00:16:33 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-b[40010]: ts=2024-09-17T00:16:33.935Z caller=node_exporter.go:115 level=info collector=textfile 2024-09-17T00:16:34.221 INFO:journalctl@ceph.node-exporter.b.smithi099.stdout:Sep 17 00:16:33 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-b[40010]: ts=2024-09-17T00:16:33.935Z caller=node_exporter.go:115 level=info collector=thermal_zone 2024-09-17T00:16:34.221 INFO:journalctl@ceph.node-exporter.b.smithi099.stdout:Sep 17 00:16:33 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-b[40010]: ts=2024-09-17T00:16:33.935Z caller=node_exporter.go:115 level=info collector=time 2024-09-17T00:16:34.221 INFO:journalctl@ceph.node-exporter.b.smithi099.stdout:Sep 17 00:16:33 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-b[40010]: ts=2024-09-17T00:16:33.935Z caller=node_exporter.go:115 level=info collector=udp_queues 2024-09-17T00:16:34.221 INFO:journalctl@ceph.node-exporter.b.smithi099.stdout:Sep 17 00:16:33 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-b[40010]: ts=2024-09-17T00:16:33.935Z caller=node_exporter.go:115 level=info collector=uname 2024-09-17T00:16:34.221 INFO:journalctl@ceph.node-exporter.b.smithi099.stdout:Sep 17 00:16:33 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-b[40010]: ts=2024-09-17T00:16:33.935Z caller=node_exporter.go:115 level=info collector=vmstat 2024-09-17T00:16:34.221 INFO:journalctl@ceph.node-exporter.b.smithi099.stdout:Sep 17 00:16:33 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-b[40010]: ts=2024-09-17T00:16:33.935Z caller=node_exporter.go:115 level=info collector=xfs 2024-09-17T00:16:34.221 INFO:journalctl@ceph.node-exporter.b.smithi099.stdout:Sep 17 00:16:33 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-b[40010]: ts=2024-09-17T00:16:33.935Z caller=node_exporter.go:115 level=info collector=zfs 2024-09-17T00:16:34.221 INFO:journalctl@ceph.node-exporter.b.smithi099.stdout:Sep 17 00:16:33 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-b[40010]: ts=2024-09-17T00:16:33.935Z caller=node_exporter.go:199 level=info msg="Listening on" address=:9100 2024-09-17T00:16:34.222 INFO:journalctl@ceph.node-exporter.b.smithi099.stdout:Sep 17 00:16:33 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-b[40010]: ts=2024-09-17T00:16:33.935Z caller=tls_config.go:195 level=info msg="TLS is disabled." http2=false 2024-09-17T00:16:34.222 INFO:journalctl@ceph.node-exporter.b.smithi099.stdout:Sep 17 00:16:34 smithi099 bash[39933]: 352ae73fe2a7d92d222a4b36681d326961a29b84383478e8bc975f417e124933 2024-09-17T00:16:34.222 INFO:journalctl@ceph.node-exporter.b.smithi099.stdout:Sep 17 00:16:34 smithi099 systemd[1]: Started Ceph node-exporter.b for 3aad27a2-7489-11ef-bceb-c7b262605968. 2024-09-17T00:16:35.217 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:16:34 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[25772]: ::ffff:172.21.15.99 - - [17/Sep/2024:00:16:34] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-09-17T00:16:35.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:16:35 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:16:35.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:16:35 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T00:16:35.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:16:35 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:16:35.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:16:35 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:16:35.416 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:16:35 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:16:35.417 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:16:35 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T00:16:35.417 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:16:35 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:16:35.417 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:16:35 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:16:35.417 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:16:35 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:16:35.417 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:16:35 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T00:16:35.417 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:16:35 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:16:35.417 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:16:35 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:16:35.418 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-17T00:16:35.418 INFO:teuthology.orchestra.run.smithi002.stdout:{"epoch":60,"fsid":"3aad27a2-7489-11ef-bceb-c7b262605968","created":"2024-09-17T00:10:48.442986+0000","modified":"2024-09-17T00:16:09.587286+0000","last_up_change":"2024-09-17T00:15:50.239941+0000","last_in_change":"2024-09-17T00:15:32.949068+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-17T00:13:54.650037+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"2030f6f2-3659-4b78-bca5-b74d15218c34","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6802","nonce":2656745758},{"type":"v1","addr":"172.21.15.2:6803","nonce":2656745758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6804","nonce":2656745758},{"type":"v1","addr":"172.21.15.2:6805","nonce":2656745758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6808","nonce":2656745758},{"type":"v1","addr":"172.21.15.2:6809","nonce":2656745758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6806","nonce":2656745758},{"type":"v1","addr":"172.21.15.2:6807","nonce":2656745758}]},"public_addr":"172.21.15.2:6803/2656745758","cluster_addr":"172.21.15.2:6805/2656745758","heartbeat_back_addr":"172.21.15.2:6809/2656745758","heartbeat_front_addr":"172.21.15.2:6807/2656745758","state":["exists","up"]},{"osd":1,"uuid":"e3883c47-fd8d-444f-8057-25d08b59af31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6810","nonce":981024217},{"type":"v1","addr":"172.21.15.2:6811","nonce":981024217}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6812","nonce":981024217},{"type":"v1","addr":"172.21.15.2:6813","nonce":981024217}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6816","nonce":981024217},{"type":"v1","addr":"172.21.15.2:6817","nonce":981024217}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6814","nonce":981024217},{"type":"v1","addr":"172.21.15.2:6815","nonce":981024217}]},"public_addr":"172.21.15.2:6811/981024217","cluster_addr":"172.21.15.2:6813/981024217","heartbeat_back_addr":"172.21.15.2:6817/981024217","heartbeat_front_addr":"172.21.15.2:6815/981024217","state":["exists","up"]},{"osd":2,"uuid":"9d042533-63e5-4f38-a8db-10b61e9b39d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6818","nonce":2219840596},{"type":"v1","addr":"172.21.15.2:6819","nonce":2219840596}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6820","nonce":2219840596},{"type":"v1","addr":"172.21.15.2:6821","nonce":2219840596}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6824","nonce":2219840596},{"type":"v1","addr":"172.21.15.2:6825","nonce":2219840596}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6822","nonce":2219840596},{"type":"v1","addr":"172.21.15.2:6823","nonce":2219840596}]},"public_addr":"172.21.15.2:6819/2219840596","cluster_addr":"172.21.15.2:6821/2219840596","heartbeat_back_addr":"172.21.15.2:6825/2219840596","heartbeat_front_addr":"172.21.15.2:6823/2219840596","state":["exists","up"]},{"osd":3,"uuid":"d9475026-f3ce-4b45-b5de-780c78dbc233","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6826","nonce":1628413766},{"type":"v1","addr":"172.21.15.2:6827","nonce":1628413766}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6828","nonce":1628413766},{"type":"v1","addr":"172.21.15.2:6829","nonce":1628413766}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6832","nonce":1628413766},{"type":"v1","addr":"172.21.15.2:6833","nonce":1628413766}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.2:6830","nonce":1628413766},{"type":"v1","addr":"172.21.15.2:6831","nonce":1628413766}]},"public_addr":"172.21.15.2:6827/1628413766","cluster_addr":"172.21.15.2:6829/1628413766","heartbeat_back_addr":"172.21.15.2:6833/1628413766","heartbeat_front_addr":"172.21.15.2:6831/1628413766","state":["exists","up"]},{"osd":4,"uuid":"f8f0575a-3699-4e8f-b969-e4b6973708df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6800","nonce":2601657006},{"type":"v1","addr":"172.21.15.99:6801","nonce":2601657006}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6802","nonce":2601657006},{"type":"v1","addr":"172.21.15.99:6803","nonce":2601657006}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6806","nonce":2601657006},{"type":"v1","addr":"172.21.15.99:6807","nonce":2601657006}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6804","nonce":2601657006},{"type":"v1","addr":"172.21.15.99:6805","nonce":2601657006}]},"public_addr":"172.21.15.99:6801/2601657006","cluster_addr":"172.21.15.99:6803/2601657006","heartbeat_back_addr":"172.21.15.99:6807/2601657006","heartbeat_front_addr":"172.21.15.99:6805/2601657006","state":["exists","up"]},{"osd":5,"uuid":"c0ba894b-bcbd-40e5-b00d-73dbddc87d59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6808","nonce":1930951601},{"type":"v1","addr":"172.21.15.99:6809","nonce":1930951601}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6810","nonce":1930951601},{"type":"v1","addr":"172.21.15.99:6811","nonce":1930951601}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6814","nonce":1930951601},{"type":"v1","addr":"172.21.15.99:6815","nonce":1930951601}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6812","nonce":1930951601},{"type":"v1","addr":"172.21.15.99:6813","nonce":1930951601}]},"public_addr":"172.21.15.99:6809/1930951601","cluster_addr":"172.21.15.99:6811/1930951601","heartbeat_back_addr":"172.21.15.99:6815/1930951601","heartbeat_front_addr":"172.21.15.99:6813/1930951601","state":["exists","up"]},{"osd":6,"uuid":"9aeb407a-4fd9-415c-aa66-1e66d8aaa700","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6816","nonce":462153986},{"type":"v1","addr":"172.21.15.99:6817","nonce":462153986}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6818","nonce":462153986},{"type":"v1","addr":"172.21.15.99:6819","nonce":462153986}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6822","nonce":462153986},{"type":"v1","addr":"172.21.15.99:6823","nonce":462153986}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6820","nonce":462153986},{"type":"v1","addr":"172.21.15.99:6821","nonce":462153986}]},"public_addr":"172.21.15.99:6817/462153986","cluster_addr":"172.21.15.99:6819/462153986","heartbeat_back_addr":"172.21.15.99:6823/462153986","heartbeat_front_addr":"172.21.15.99:6821/462153986","state":["exists","up"]},{"osd":7,"uuid":"9b87a926-31b4-4937-b425-b52e773f79e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6824","nonce":544554822},{"type":"v1","addr":"172.21.15.99:6825","nonce":544554822}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6826","nonce":544554822},{"type":"v1","addr":"172.21.15.99:6827","nonce":544554822}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6830","nonce":544554822},{"type":"v1","addr":"172.21.15.99:6831","nonce":544554822}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.99:6828","nonce":544554822},{"type":"v1","addr":"172.21.15.99:6829","nonce":544554822}]},"public_addr":"172.21.15.99:6825/544554822","cluster_addr":"172.21.15.99:6827/544554822","heartbeat_back_addr":"172.21.15.99:6831/544554822","heartbeat_front_addr":"172.21.15.99:6829/544554822","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T00:13:02.874193+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T00:13:28.064593+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T00:13:51.522483+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T00:14:15.866757+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T00:14:37.409107+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T00:15:00.424239+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T00:15:24.796694+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-09-17T00:15:48.372673+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.2:0/1056079102":"2024-09-18T00:16:09.587221+0000","172.21.15.2:6801/3807888742":"2024-09-18T00:16:09.587221+0000","172.21.15.2:6800/3807888742":"2024-09-18T00:16:09.587221+0000","172.21.15.2:0/1754068649":"2024-09-18T00:16:09.587221+0000","172.21.15.2:0/1780251331":"2024-09-18T00:11:44.094172+0000","172.21.15.2:0/1474587242":"2024-09-18T00:16:09.587221+0000","172.21.15.2:0/824729147":"2024-09-18T00:11:44.094172+0000","172.21.15.2:6801/2522360616":"2024-09-18T00:11:44.094172+0000","172.21.15.2:6800/2522360616":"2024-09-18T00:11:44.094172+0000","172.21.15.2:0/3091500665":"2024-09-18T00:11:44.094172+0000","172.21.15.2:0/3536227112":"2024-09-18T00:11:20.938701+0000","172.21.15.2:6800/3667297605":"2024-09-18T00:11:20.938701+0000","172.21.15.2:0/626695359":"2024-09-18T00:16:09.587221+0000","172.21.15.2:6801/3667297605":"2024-09-18T00:11:20.938701+0000","172.21.15.2:0/3867827864":"2024-09-18T00:11:20.938701+0000","172.21.15.2:0/174032459":"2024-09-18T00:11:20.938701+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-17T00:16:36.188 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:16:36 smithi002 ceph-mon[28203]: pgmap v15: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:16:36.188 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:16:36 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:16:36.188 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:16:36 smithi002 ceph-mon[28203]: from='client.? 172.21.15.2:0/1567011722' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-17T00:16:36.188 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:16:36 smithi002 ceph-mon[23737]: pgmap v15: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:16:36.188 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:16:36 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:16:36.189 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:16:36 smithi002 ceph-mon[23737]: from='client.? 172.21.15.2:0/1567011722' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-17T00:16:36.213 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 3aad27a2-7489-11ef-bceb-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-09-17T00:16:36.213 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 3aad27a2-7489-11ef-bceb-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-09-17T00:16:36.214 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 3aad27a2-7489-11ef-bceb-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-09-17T00:16:36.214 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 3aad27a2-7489-11ef-bceb-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-09-17T00:16:36.214 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 3aad27a2-7489-11ef-bceb-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-09-17T00:16:36.215 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 3aad27a2-7489-11ef-bceb-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-09-17T00:16:36.215 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 3aad27a2-7489-11ef-bceb-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-09-17T00:16:36.216 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 3aad27a2-7489-11ef-bceb-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-09-17T00:16:36.358 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:16:36 smithi099 ceph-mon[24594]: pgmap v15: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:16:36.358 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:16:36 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:16:36.358 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:16:36 smithi099 ceph-mon[24594]: from='client.? 172.21.15.2:0/1567011722' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-17T00:16:36.778 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T00:16:36.787 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T00:16:36.792 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T00:16:36.793 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T00:16:36.797 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T00:16:36.800 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T00:16:36.826 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T00:16:36.857 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T00:16:37.816 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:16:37 smithi002 ceph-mon[28203]: pgmap v16: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:16:37.816 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:16:37 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:16:37.817 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:16:37 smithi002 ceph-mon[23737]: pgmap v16: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:16:37.817 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:16:37 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:16:37.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:16:37 smithi099 ceph-mon[24594]: pgmap v16: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:16:37.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:16:37 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:16:39.229 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:16:39 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: ::ffff:172.21.15.99 - - [17/Sep/2024:00:16:39] "GET /metrics HTTP/1.1" 200 191488 "" "Prometheus/2.33.4" 2024-09-17T00:16:39.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:16:39 smithi099 ceph-mon[24594]: pgmap v17: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:16:39.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:16:39 smithi002 ceph-mon[23737]: pgmap v17: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:16:39.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:16:39 smithi002 ceph-mon[28203]: pgmap v17: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:16:41.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:16:41 smithi002 ceph-mon[23737]: pgmap v18: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:16:41.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:16:41 smithi002 ceph-mon[28203]: pgmap v18: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:16:41.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:16:41 smithi099 ceph-mon[24594]: pgmap v18: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:16:43.662 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:16:43 smithi002 ceph-mon[28203]: pgmap v19: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:16:43.663 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:16:43 smithi002 ceph-mon[23737]: pgmap v19: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:16:43.966 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:16:43 smithi099 ceph-mon[24594]: pgmap v19: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:16:45.217 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:16:44 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[25772]: ::ffff:172.21.15.99 - - [17/Sep/2024:00:16:44] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-09-17T00:16:45.596 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:16:45 smithi002 ceph-mon[23737]: pgmap v20: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:16:45.597 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:16:45 smithi002 ceph-mon[28203]: pgmap v20: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:16:45.966 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:16:45 smithi099 ceph-mon[24594]: pgmap v20: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:16:46.778 INFO:teuthology.orchestra.run.smithi002.stdout:55834574888 2024-09-17T00:16:46.778 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 3aad27a2-7489-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-09-17T00:16:47.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:16:47 smithi002 ceph-mon[28203]: pgmap v21: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:16:47.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:16:47 smithi002 ceph-mon[23737]: pgmap v21: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:16:47.966 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:16:47 smithi099 ceph-mon[24594]: pgmap v21: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:16:48.103 INFO:teuthology.orchestra.run.smithi002.stdout:77309411364 2024-09-17T00:16:48.103 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 3aad27a2-7489-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-09-17T00:16:49.229 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:16:49 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: ::ffff:172.21.15.99 - - [17/Sep/2024:00:16:49] "GET /metrics HTTP/1.1" 200 191474 "" "Prometheus/2.33.4" 2024-09-17T00:16:49.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:16:49 smithi099 ceph-mon[24594]: pgmap v22: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:16:49.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:16:49 smithi002 ceph-mon[28203]: pgmap v22: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:16:49.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:16:49 smithi002 ceph-mon[23737]: pgmap v22: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:16:50.910 INFO:teuthology.orchestra.run.smithi002.stdout:111669149728 2024-09-17T00:16:50.910 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 3aad27a2-7489-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-09-17T00:16:51.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:16:51 smithi099 ceph-mon[24594]: pgmap v23: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:16:51.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:16:51 smithi002 ceph-mon[28203]: pgmap v23: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:16:51.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:16:51 smithi002 ceph-mon[23737]: pgmap v23: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:16:52.477 INFO:teuthology.orchestra.run.smithi002.stdout:167503724567 2024-09-17T00:16:52.477 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 3aad27a2-7489-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-09-17T00:16:53.224 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:16:53 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:16:53.224 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:16:53 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:16:53.224 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:16:53 smithi002 ceph-mon[28203]: Deploying daemon alertmanager.a on smithi002 2024-09-17T00:16:53.225 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:16:53 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:16:53.225 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:16:53 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:16:53.225 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:16:53 smithi002 ceph-mon[23737]: Deploying daemon alertmanager.a on smithi002 2024-09-17T00:16:53.246 INFO:teuthology.orchestra.run.smithi002.stdout:133143986202 2024-09-17T00:16:53.247 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 3aad27a2-7489-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-09-17T00:16:53.466 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:16:53 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:16:53.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:16:53 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:16:53.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:16:53 smithi099 ceph-mon[24594]: Deploying daemon alertmanager.a on smithi002 2024-09-17T00:16:53.704 INFO:teuthology.orchestra.run.smithi002.stdout:34359738415 2024-09-17T00:16:53.704 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 3aad27a2-7489-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-09-17T00:16:54.145 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T00:16:54.145 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T00:16:54.146 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T00:16:54.178 INFO:teuthology.orchestra.run.smithi002.stdout:236223201291 2024-09-17T00:16:54.178 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 3aad27a2-7489-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-09-17T00:16:54.314 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:16:54 smithi002 ceph-mon[28203]: pgmap v24: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:16:54.314 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:16:54 smithi002 ceph-mon[23737]: pgmap v24: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:16:54.368 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:16:54 smithi099 ceph-mon[24594]: pgmap v24: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:16:55.221 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:16:54 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[25772]: ::ffff:172.21.15.99 - - [17/Sep/2024:00:16:54] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-09-17T00:16:56.338 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:16:56 smithi002 ceph-mon[28203]: pgmap v25: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:16:56.338 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:16:56 smithi002 ceph-mon[23737]: pgmap v25: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:16:56.466 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:16:56 smithi099 ceph-mon[24594]: pgmap v25: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:16:57.200 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T00:16:57.200 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T00:16:57.206 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T00:16:57.208 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T00:16:57.376 INFO:teuthology.orchestra.run.smithi002.stdout:111669149729 2024-09-17T00:16:57.716 INFO:teuthology.orchestra.run.smithi002.stdout:201863462932 2024-09-17T00:16:57.717 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 3aad27a2-7489-11ef-bceb-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-09-17T00:16:58.466 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:16:58 smithi099 ceph-mon[24594]: pgmap v26: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:16:58.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:16:58 smithi099 ceph-mon[24594]: from='client.? 172.21.15.2:0/3187371513' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-17T00:16:58.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:16:58 smithi002 ceph-mon[23737]: pgmap v26: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:16:58.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:16:58 smithi002 ceph-mon[23737]: from='client.? 172.21.15.2:0/3187371513' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-17T00:16:58.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:16:58 smithi002 ceph-mon[28203]: pgmap v26: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:16:58.480 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:16:58 smithi002 ceph-mon[28203]: from='client.? 172.21.15.2:0/3187371513' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-17T00:16:59.185 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:16:59 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: ::ffff:172.21.15.99 - - [17/Sep/2024:00:16:59] "GET /metrics HTTP/1.1" 200 191461 "" "Prometheus/2.33.4" 2024-09-17T00:16:59.914 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149728 got 111669149729 for osd.3 2024-09-17T00:16:59.915 DEBUG:teuthology.parallel:result is None 2024-09-17T00:17:00.223 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:17:00 smithi002 ceph-mon[23737]: pgmap v27: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:17:00.224 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:17:00 smithi002 ceph-mon[23737]: from='client.? 172.21.15.2:0/2215179207' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-17T00:17:00.224 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:17:00 smithi002 ceph-mon[28203]: pgmap v27: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:17:00.224 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:17:00 smithi002 ceph-mon[28203]: from='client.? 172.21.15.2:0/2215179207' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-17T00:17:00.343 INFO:teuthology.orchestra.run.smithi002.stdout:55834574891 2024-09-17T00:17:00.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:17:00 smithi099 ceph-mon[24594]: pgmap v27: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:17:00.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:17:00 smithi099 ceph-mon[24594]: from='client.? 172.21.15.2:0/2215179207' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-17T00:17:02.337 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:17:02 smithi002 ceph-mon[28203]: pgmap v28: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:17:02.337 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:17:02 smithi002 ceph-mon[23737]: pgmap v28: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:17:02.466 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:17:02 smithi099 ceph-mon[24594]: pgmap v28: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:17:04.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:17:04 smithi099 ceph-mon[24594]: pgmap v29: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:17:04.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:17:04 smithi002 ceph-mon[23737]: pgmap v29: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:17:04.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:17:04 smithi002 ceph-mon[28203]: pgmap v29: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:17:04.627 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574888 got 55834574891 for osd.1 2024-09-17T00:17:04.628 DEBUG:teuthology.parallel:result is None 2024-09-17T00:17:05.217 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:17:04 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[25772]: ::ffff:172.21.15.99 - - [17/Sep/2024:00:17:04] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-09-17T00:17:05.552 INFO:teuthology.orchestra.run.smithi002.stdout:77309411368 2024-09-17T00:17:06.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:17:06 smithi099 ceph-mon[24594]: pgmap v30: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:17:06.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:17:06 smithi099 ceph-mon[24594]: from='client.? 172.21.15.2:0/4136951474' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-17T00:17:06.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:17:06 smithi002 ceph-mon[28203]: pgmap v30: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:17:06.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:17:06 smithi002 ceph-mon[28203]: from='client.? 172.21.15.2:0/4136951474' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-17T00:17:06.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:17:06 smithi002 ceph-mon[23737]: pgmap v30: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:17:06.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:17:06 smithi002 ceph-mon[23737]: from='client.? 172.21.15.2:0/4136951474' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-17T00:17:06.538 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411364 got 77309411368 for osd.2 2024-09-17T00:17:06.538 DEBUG:teuthology.parallel:result is None 2024-09-17T00:17:06.806 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T00:17:07.962 INFO:teuthology.orchestra.run.smithi002.stdout:236223201295 2024-09-17T00:17:08.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:17:08 smithi002 ceph-mon[28203]: pgmap v31: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:17:08.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:17:08 smithi002 ceph-mon[28203]: from='client.? 172.21.15.2:0/3578069555' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-17T00:17:08.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:17:08 smithi002 ceph-mon[23737]: pgmap v31: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:17:08.231 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:17:08 smithi002 ceph-mon[23737]: from='client.? 172.21.15.2:0/3578069555' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-17T00:17:08.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:17:08 smithi099 ceph-mon[24594]: pgmap v31: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:17:08.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:17:08 smithi099 ceph-mon[24594]: from='client.? 172.21.15.2:0/3578069555' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-17T00:17:09.077 INFO:teuthology.orchestra.run.smithi002.stdout:167503724571 2024-09-17T00:17:09.270 INFO:teuthology.orchestra.run.smithi002.stdout:133143986207 2024-09-17T00:17:09.302 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:17:09 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: ::ffff:172.21.15.99 - - [17/Sep/2024:00:17:09] "GET /metrics HTTP/1.1" 200 191461 "" "Prometheus/2.33.4" 2024-09-17T00:17:10.373 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:17:10 smithi002 ceph-mon[28203]: pgmap v32: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:17:10.373 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:17:10 smithi002 ceph-mon[28203]: from='client.? 172.21.15.2:0/1163207250' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-17T00:17:10.373 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:17:10 smithi002 ceph-mon[28203]: from='client.? 172.21.15.2:0/2830610434' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-17T00:17:10.373 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:17:10 smithi002 ceph-mon[23737]: pgmap v32: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:17:10.373 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:17:10 smithi002 ceph-mon[23737]: from='client.? 172.21.15.2:0/1163207250' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-17T00:17:10.373 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:17:10 smithi002 ceph-mon[23737]: from='client.? 172.21.15.2:0/2830610434' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-17T00:17:10.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:17:10 smithi099 ceph-mon[24594]: pgmap v32: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:17:10.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:17:10 smithi099 ceph-mon[24594]: from='client.? 172.21.15.2:0/1163207250' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-17T00:17:10.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:17:10 smithi099 ceph-mon[24594]: from='client.? 172.21.15.2:0/2830610434' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-17T00:17:11.396 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:17:11 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T00:17:11.396 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:17:11 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T00:17:11.396 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:17:11 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T00:17:11.396 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:17:11 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T00:17:11.466 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:17:11 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T00:17:11.466 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:17:11 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T00:17:12.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:17:12 smithi002 ceph-mon[23737]: pgmap v33: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:17:12.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:17:12 smithi002 ceph-mon[28203]: pgmap v33: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:17:12.322 INFO:tasks.cephadm.ceph_manager.ceph:need seq 236223201291 got 236223201295 for osd.7 2024-09-17T00:17:12.323 DEBUG:teuthology.parallel:result is None 2024-09-17T00:17:12.466 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:17:12 smithi099 ceph-mon[24594]: pgmap v33: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:17:13.112 INFO:teuthology.orchestra.run.smithi002.stdout:34359738420 2024-09-17T00:17:14.016 INFO:tasks.cephadm.ceph_manager.ceph:need seq 133143986202 got 133143986207 for osd.4 2024-09-17T00:17:14.016 DEBUG:teuthology.parallel:result is None 2024-09-17T00:17:14.238 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:17:14 smithi002 ceph-mon[28203]: pgmap v34: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:17:14.238 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:17:14 smithi002 ceph-mon[28203]: from='client.? 172.21.15.2:0/468639722' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-17T00:17:14.239 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:17:14 smithi002 ceph-mon[23737]: pgmap v34: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:17:14.239 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:17:14 smithi002 ceph-mon[23737]: from='client.? 172.21.15.2:0/468639722' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-17T00:17:14.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:17:14 smithi099 ceph-mon[24594]: pgmap v34: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:17:14.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:17:14 smithi099 ceph-mon[24594]: from='client.? 172.21.15.2:0/468639722' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-17T00:17:15.217 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:17:14 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[25772]: ::ffff:172.21.15.99 - - [17/Sep/2024:00:17:14] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-09-17T00:17:15.396 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738415 got 34359738420 for osd.0 2024-09-17T00:17:15.396 DEBUG:teuthology.parallel:result is None 2024-09-17T00:17:15.792 INFO:teuthology.orchestra.run.smithi002.stdout:201863462936 2024-09-17T00:17:15.955 INFO:tasks.cephadm.ceph_manager.ceph:need seq 167503724567 got 167503724571 for osd.5 2024-09-17T00:17:15.955 DEBUG:teuthology.parallel:result is None 2024-09-17T00:17:16.466 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:17:16 smithi099 ceph-mon[24594]: pgmap v35: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:17:16.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:17:16 smithi099 ceph-mon[24594]: from='client.? 172.21.15.2:0/2752975933' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-17T00:17:16.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:17:16 smithi002 ceph-mon[23737]: pgmap v35: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:17:16.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:17:16 smithi002 ceph-mon[23737]: from='client.? 172.21.15.2:0/2752975933' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-17T00:17:16.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:17:16 smithi002 ceph-mon[28203]: pgmap v35: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:17:16.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:17:16 smithi002 ceph-mon[28203]: from='client.? 172.21.15.2:0/2752975933' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-17T00:17:17.479 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Sep 17 00:17:17 smithi002 systemd[1]: Starting Ceph alertmanager.a for 3aad27a2-7489-11ef-bceb-c7b262605968... 2024-09-17T00:17:17.522 INFO:tasks.cephadm.ceph_manager.ceph:need seq 201863462932 got 201863462936 for osd.6 2024-09-17T00:17:17.523 DEBUG:teuthology.parallel:result is None 2024-09-17T00:17:17.523 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-09-17T00:17:17.523 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 3aad27a2-7489-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-17T00:17:17.825 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Sep 17 00:17:17 smithi002 podman[46236]: 2024-09-17 00:17:17.736797684 +0000 UTC m=+0.112730967 volume create f4329900baca96a599cf7e73b3234db1b86b382c8d6d8de42f9cf1bed00cc46d 2024-09-17T00:17:17.825 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Sep 17 00:17:17 smithi002 podman[46236]: 2024-09-17 00:17:17.64080218 +0000 UTC m=+0.016735397 image pull ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4 quay.io/prometheus/alertmanager:v0.23.0 2024-09-17T00:17:17.825 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Sep 17 00:17:17 smithi002 podman[46236]: 2024-09-17 00:17:17.82865779 +0000 UTC m=+0.204591004 container create 7c35d04fcfe4066f9a6d8928d1a58dd6239e09ef450878894772070b9f00283b (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-17T00:17:18.102 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:17:18 smithi002 ceph-mon[28203]: pgmap v36: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:17:18.103 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:17:18 smithi002 ceph-mon[23737]: pgmap v36: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:17:18.283 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T00:17:18.388 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:17:18 smithi099 ceph-mon[24594]: pgmap v36: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:17:18.406 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Sep 17 00:17:18 smithi002 podman[46236]: 2024-09-17 00:17:18.106452699 +0000 UTC m=+0.482385921 container init 7c35d04fcfe4066f9a6d8928d1a58dd6239e09ef450878894772070b9f00283b (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-17T00:17:18.406 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Sep 17 00:17:18 smithi002 podman[46236]: 2024-09-17 00:17:18.109907175 +0000 UTC m=+0.485840399 container start 7c35d04fcfe4066f9a6d8928d1a58dd6239e09ef450878894772070b9f00283b (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-17T00:17:18.406 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Sep 17 00:17:18 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-alertmanager-a[46262]: level=info ts=2024-09-17T00:17:18.121Z caller=main.go:225 msg="Starting Alertmanager" version="(version=0.23.0, branch=HEAD, revision=61046b17771a57cfd4c4a51be370ab930a4d7d54)" 2024-09-17T00:17:18.406 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Sep 17 00:17:18 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-alertmanager-a[46262]: level=info ts=2024-09-17T00:17:18.121Z caller=main.go:226 build_context="(go=go1.16.7, user=root@e21a959be8d2, date=20210825-10:48:55)" 2024-09-17T00:17:18.406 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Sep 17 00:17:18 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-alertmanager-a[46262]: level=info ts=2024-09-17T00:17:18.122Z caller=cluster.go:184 component=cluster msg="setting advertise address explicitly" addr=172.21.15.2 port=9094 2024-09-17T00:17:18.406 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Sep 17 00:17:18 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-alertmanager-a[46262]: level=info ts=2024-09-17T00:17:18.123Z caller=cluster.go:671 component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-09-17T00:17:18.406 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Sep 17 00:17:18 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-alertmanager-a[46262]: level=info ts=2024-09-17T00:17:18.148Z caller=coordinator.go:113 component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-17T00:17:18.406 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Sep 17 00:17:18 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-alertmanager-a[46262]: level=info ts=2024-09-17T00:17:18.148Z caller=coordinator.go:126 component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-17T00:17:18.406 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Sep 17 00:17:18 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-alertmanager-a[46262]: level=info ts=2024-09-17T00:17:18.151Z caller=main.go:518 msg=Listening address=:9093 2024-09-17T00:17:18.407 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Sep 17 00:17:18 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-alertmanager-a[46262]: level=info ts=2024-09-17T00:17:18.151Z caller=tls_config.go:191 msg="TLS is disabled." http2=false 2024-09-17T00:17:18.407 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Sep 17 00:17:18 smithi002 bash[46236]: 7c35d04fcfe4066f9a6d8928d1a58dd6239e09ef450878894772070b9f00283b 2024-09-17T00:17:18.407 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Sep 17 00:17:18 smithi002 systemd[1]: Started Ceph alertmanager.a for 3aad27a2-7489-11ef-bceb-c7b262605968. 2024-09-17T00:17:19.409 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:17:19 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: ::ffff:172.21.15.99 - - [17/Sep/2024:00:17:19] "GET /metrics HTTP/1.1" 200 191464 "" "Prometheus/2.33.4" 2024-09-17T00:17:19.410 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:17:19 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:17:19.410 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:17:19 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:17:19.410 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:17:19 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:17:19.410 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:17:19 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-17T00:17:19.410 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:17:19 smithi002 ceph-mon[23737]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-17T00:17:19.410 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:17:19 smithi002 ceph-mon[23737]: pgmap v37: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:17:19.410 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:17:19 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:17:19.410 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:17:19 smithi002 ceph-mon[23737]: Deploying daemon grafana.a on smithi099 2024-09-17T00:17:19.410 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:17:19 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:17:19.410 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:17:19 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:17:19.410 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:17:19 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:17:19.411 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:17:19 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-17T00:17:19.411 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:17:19 smithi002 ceph-mon[28203]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-17T00:17:19.411 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:17:19 smithi002 ceph-mon[28203]: pgmap v37: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:17:19.411 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:17:19 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:17:19.411 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:17:19 smithi002 ceph-mon[28203]: Deploying daemon grafana.a on smithi099 2024-09-17T00:17:19.716 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:17:19 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:17:19.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:17:19 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:17:19.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:17:19 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:17:19.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:17:19 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-17T00:17:19.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:17:19 smithi099 ceph-mon[24594]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-17T00:17:19.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:17:19 smithi099 ceph-mon[24594]: pgmap v37: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:17:19.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:17:19 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:17:19.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:17:19 smithi099 ceph-mon[24594]: Deploying daemon grafana.a on smithi099 2024-09-17T00:17:20.114 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-17T00:17:20.116 INFO:teuthology.orchestra.run.smithi002.stderr:dumped all 2024-09-17T00:17:20.424 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Sep 17 00:17:20 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-alertmanager-a[46262]: level=info ts=2024-09-17T00:17:20.124Z caller=cluster.go:696 component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.00105473s 2024-09-17T00:17:20.424 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:17:20 smithi002 ceph-mon[28203]: from='client.24496 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T00:17:20.424 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:17:20 smithi002 ceph-mon[23737]: from='client.24496 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T00:17:20.717 INFO:teuthology.orchestra.run.smithi002.stdout:{"pg_ready":true,"pg_map":{"version":37,"stamp":"2024-09-17T00:17:18.358931+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":749961216,"kb_used":48968,"kb_used_data":6152,"kb_used_omap":0,"kb_used_meta":42752,"kb_avail":749912248,"statfs":{"total":767960285184,"available":767910141952,"internally_reserved":0,"allocated":6299648,"data_stored":3510946,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":43778048},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002363"},"pg_stats":[{"pgid":"1.0","version":"60'128","reported_seq":267,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-17T00:16:10.433169+0000","last_change":"2024-09-17T00:15:53.556149+0000","last_active":"2024-09-17T00:16:10.433169+0000","last_peered":"2024-09-17T00:16:10.433169+0000","last_clean":"2024-09-17T00:16:10.433169+0000","last_became_active":"2024-09-17T00:15:53.555687+0000","last_became_peered":"2024-09-17T00:15:53.555687+0000","last_unstale":"2024-09-17T00:16:10.433169+0000","last_undegraded":"2024-09-17T00:16:10.433169+0000","last_fullsized":"2024-09-17T00:16:10.433169+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T00:13:54.787965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T00:13:54.787965+0000","last_clean_scrub_stamp":"2024-09-17T00:13:54.787965+0000","objects_scrubbed":0,"log_size":128,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:30:48.289980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":55,"seq":236223201298,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6096,"kb_used_data":1096,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":93739056,"statfs":{"total":95995035648,"available":95988793344,"internally_reserved":0,"allocated":1122304,"data_stored":769588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 00:16:50 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.501,"15min":0.501},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.376}]},{"osd":1,"last update":"Tue Sep 17 00:16:50 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.48799999999999999}]},{"osd":2,"last update":"Tue Sep 17 00:16:50 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.60499999999999998}]},{"osd":3,"last update":"Tue Sep 17 00:16:50 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.871,"5min":0.871,"15min":0.871},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.47399999999999998}]},{"osd":4,"last update":"Tue Sep 17 00:16:50 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.62}]},{"osd":5,"last update":"Tue Sep 17 00:16:50 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.504,"5min":0.504,"15min":0.504},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.624}]},{"osd":6,"last update":"Tue Sep 17 00:16:50 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.58299999999999996}]}]},{"osd":6,"up_from":47,"seq":201863462937,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6152,"kb_used_data":1088,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93739000,"statfs":{"total":95995035648,"available":95988736000,"internally_reserved":0,"allocated":1114112,"data_stored":769000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 00:16:30 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.48499999999999999}]},{"osd":1,"last update":"Tue Sep 17 00:16:30 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.56299999999999994}]},{"osd":2,"last update":"Tue Sep 17 00:16:30 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.54200000000000004}]},{"osd":3,"last update":"Tue Sep 17 00:16:30 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.52900000000000003}]},{"osd":4,"last update":"Tue Sep 17 00:16:30 2024","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"last":0.41199999999999998}]},{"osd":5,"last update":"Tue Sep 17 00:16:30 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.58799999999999997}]},{"osd":7,"last update":"Tue Sep 17 00:16:50 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.434},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.57599999999999996}]}]},{"osd":1,"up_from":13,"seq":55834574895,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6344,"kb_used_data":576,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738808,"statfs":{"total":95995035648,"available":95988539392,"internally_reserved":0,"allocated":589824,"data_stored":240660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 00:16:35 2024","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.34599999999999997,"15min":0.32600000000000001},"min":{"1min":0.247,"5min":0.214,"15min":0.214},"max":{"1min":0.59099999999999997,"5min":0.63600000000000001,"15min":0.63600000000000001},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.36599999999999999,"5min":0.36199999999999999,"15min":0.34399999999999997},"min":{"1min":0.26400000000000001,"5min":0.214,"15min":0.214},"max":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"last":0.41099999999999998}]},{"osd":2,"last update":"Tue Sep 17 00:16:58 2024","interfaces":[{"interface":"back","average":{"1min":0.38,"5min":0.374,"15min":0.36399999999999999},"min":{"1min":0.26800000000000002,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":1.01,"5min":1.01,"15min":1.01},"last":0.376},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.40699999999999997,"15min":0.39400000000000002},"min":{"1min":0.30499999999999999,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.39000000000000001}]},{"osd":3,"last update":"Tue Sep 17 00:16:22 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.48899999999999999,"15min":0.49099999999999999},"min":{"1min":0.35999999999999999,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.47899999999999998}]},{"osd":4,"last update":"Tue Sep 17 00:16:41 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.60599999999999998,"15min":0.61399999999999999},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.93100000000000005,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.61199999999999999,"15min":0.62},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.60199999999999998}]},{"osd":5,"last update":"Tue Sep 17 00:17:06 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.59699999999999998,"15min":0.60199999999999998},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.53600000000000003},{"interface":"front","average":{"1min":1.0960000000000001,"5min":0.73599999999999999,"15min":0.67600000000000005},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":11.391999999999999,"5min":11.391999999999999,"15min":11.391999999999999},"last":0.61399999999999999}]},{"osd":6,"last update":"Tue Sep 17 00:16:26 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.626}]},{"osd":7,"last update":"Tue Sep 17 00:16:52 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.044,"5min":1.044,"15min":1.044},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.54900000000000004}]}]},{"osd":0,"up_from":8,"seq":34359738421,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6928,"kb_used_data":1096,"kb_used_omap":0,"kb_used_meta":5824,"kb_avail":93738224,"statfs":{"total":95995035648,"available":95987941376,"internally_reserved":0,"allocated":1122304,"data_stored":769588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5963776},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Sep 17 00:16:39 2024","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.36199999999999999,"15min":0.34499999999999997},"min":{"1min":0.27000000000000002,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.41999999999999998,"5min":0.38800000000000001,"15min":0.35999999999999999},"min":{"1min":0.30599999999999999,"5min":0.246,"15min":0.246},"max":{"1min":0.61599999999999999,"5min":0.65400000000000003,"15min":0.65400000000000003},"last":0.51100000000000001}]},{"osd":2,"last update":"Tue Sep 17 00:17:01 2024","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.40200000000000002,"15min":0.39900000000000002},"min":{"1min":0.29099999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.47899999999999998,"5min":0.64300000000000002,"15min":0.64300000000000002},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.41299999999999998,"5min":0.43099999999999999,"15min":0.42499999999999999},"min":{"1min":0.32800000000000001,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.51900000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"last":0.54000000000000004}]},{"osd":3,"last update":"Tue Sep 17 00:16:21 2024","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.45400000000000001,"15min":0.45600000000000002},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.58299999999999996,"5min":0.626,"15min":0.626},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.503,"15min":0.50600000000000001},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.53100000000000003}]},{"osd":4,"last update":"Tue Sep 17 00:16:42 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.63,"15min":0.63200000000000001},"min":{"1min":0.45500000000000002,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.629,"5min":0.64100000000000001,"15min":0.64300000000000002},"min":{"1min":0.48299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.85299999999999998,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.81100000000000005}]},{"osd":5,"last update":"Tue Sep 17 00:17:06 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.59199999999999997,"15min":0.58799999999999997},"min":{"1min":0.438,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.93100000000000005,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.82899999999999996},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.61699999999999999,"15min":0.621},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.67800000000000005}]},{"osd":6,"last update":"Tue Sep 17 00:16:27 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.623,"15min":0.623},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":1.0189999999999999,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.624,"5min":0.624,"15min":0.624},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.77600000000000002}]},{"osd":7,"last update":"Tue Sep 17 00:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.75600000000000001},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.98299999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":0.71699999999999997}]}]},{"osd":2,"up_from":18,"seq":77309411371,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6276,"kb_used_data":572,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":93738876,"statfs":{"total":95995035648,"available":95988609024,"internally_reserved":0,"allocated":585728,"data_stored":240395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 00:16:54 2024","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.37,"15min":0.35799999999999998},"min":{"1min":0.25900000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.55600000000000005,"5min":0.65100000000000002,"15min":0.65100000000000002},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.39000000000000001,"15min":0.38300000000000001},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"last":0.56599999999999995}]},{"osd":1,"last update":"Tue Sep 17 00:16:54 2024","interfaces":[{"interface":"back","average":{"1min":0.39100000000000001,"5min":0.40100000000000002,"15min":0.39300000000000002},"min":{"1min":0.28399999999999997,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.67900000000000005,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.41799999999999998,"5min":0.41699999999999998,"15min":0.40699999999999997},"min":{"1min":0.28199999999999997,"5min":0.25,"15min":0.25},"max":{"1min":0.58199999999999996,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.495}]},{"osd":3,"last update":"Tue Sep 17 00:16:23 2024","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.47999999999999998,"15min":0.48099999999999998},"min":{"1min":0.36299999999999999,"5min":0.307,"15min":0.307},"max":{"1min":0.59999999999999998,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.505,"15min":0.50900000000000001},"min":{"1min":0.317,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.629,"5min":0.67700000000000005,"15min":0.67700000000000005},"last":0.61299999999999999}]},{"osd":4,"last update":"Tue Sep 17 00:16:42 2024","interfaces":[{"interface":"back","average":{"1min":0.81699999999999995,"5min":0.625,"15min":0.59299999999999997},"min":{"1min":0.40899999999999997,"5min":0.373,"15min":0.373},"max":{"1min":7.2960000000000003,"5min":7.2960000000000003,"15min":7.2960000000000003},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.60799999999999998,"15min":0.60699999999999998},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.82899999999999996,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.66200000000000003}]},{"osd":5,"last update":"Tue Sep 17 00:17:03 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.59399999999999997,"15min":0.59999999999999998},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.77100000000000002,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.63300000000000001,"15min":0.64000000000000001},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.84999999999999998,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.68000000000000005}]},{"osd":6,"last update":"Tue Sep 17 00:16:27 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.51600000000000001}]},{"osd":7,"last update":"Tue Sep 17 00:16:50 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.76800000000000002}]}]},{"osd":3,"up_from":26,"seq":111669149734,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5700,"kb_used_data":572,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739452,"statfs":{"total":95995035648,"available":95989198848,"internally_reserved":0,"allocated":585728,"data_stored":240395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 00:16:20 2024","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.39900000000000002,"15min":0.40200000000000002},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.55200000000000005,"5min":0.57799999999999996,"15min":0.57799999999999996},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.434,"15min":0.42999999999999999},"min":{"1min":0.32500000000000001,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.25600000000000001}]},{"osd":1,"last update":"Tue Sep 17 00:16:20 2024","interfaces":[{"interface":"back","average":{"1min":0.42499999999999999,"5min":0.42899999999999999,"15min":0.42999999999999999},"min":{"1min":0.33300000000000002,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.67800000000000005,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":0.30599999999999999},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.44900000000000001,"15min":0.44700000000000001},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.28999999999999998}]},{"osd":2,"last update":"Tue Sep 17 00:16:20 2024","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.48299999999999998,"15min":0.48499999999999999},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.61799999999999999,"5min":0.64500000000000002,"15min":0.64500000000000002},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.45600000000000002,"15min":0.45700000000000002},"min":{"1min":0.33000000000000002,"5min":0.311,"15min":0.311},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.34699999999999998}]},{"osd":4,"last update":"Tue Sep 17 00:16:43 2024","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.60199999999999998,"15min":0.59599999999999997},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.58599999999999997,"15min":0.58399999999999996},"min":{"1min":0.45400000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.47499999999999998}]},{"osd":5,"last update":"Tue Sep 17 00:17:06 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.60299999999999998,"15min":0.60599999999999998},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.72899999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.622,"15min":0.626},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.82899999999999996,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.58399999999999996}]},{"osd":6,"last update":"Tue Sep 17 00:16:28 2024","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.873,"5min":0.873,"15min":0.873},"last":0.50900000000000001}]},{"osd":7,"last update":"Tue Sep 17 00:16:51 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.499},{"interface":"front","average":{"1min":0.628,"5min":0.628,"15min":0.628},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.51400000000000001}]}]},{"osd":4,"up_from":31,"seq":133143986209,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5768,"kb_used_data":576,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":93739384,"statfs":{"total":95995035648,"available":95989129216,"internally_reserved":0,"allocated":589824,"data_stored":240660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 00:16:41 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.55700000000000005,"15min":0.55100000000000005},"min":{"1min":0.41899999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.873,"5min":0.873,"15min":0.873},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.57799999999999996,"15min":0.57699999999999996},"min":{"1min":0.35299999999999998,"5min":0.318,"15min":0.318},"max":{"1min":0.83299999999999996,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.67900000000000005}]},{"osd":1,"last update":"Tue Sep 17 00:16:41 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.56200000000000006,"15min":0.56000000000000005},"min":{"1min":0.38700000000000001,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.73999999999999999},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.57499999999999996,"15min":0.56899999999999995},"min":{"1min":0.46100000000000002,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.83199999999999996}]},{"osd":2,"last update":"Tue Sep 17 00:16:41 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.57699999999999996,"15min":0.57299999999999995},"min":{"1min":0.40100000000000002,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.85399999999999998,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.80400000000000005},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.57099999999999995,"15min":0.56799999999999995},"min":{"1min":0.38500000000000001,"5min":0.374,"15min":0.374},"max":{"1min":0.79500000000000004,"5min":0.88,"15min":0.88},"last":0.75}]},{"osd":3,"last update":"Tue Sep 17 00:16:41 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.41099999999999998,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.79100000000000004},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.56100000000000005,"15min":0.55500000000000005},"min":{"1min":0.434,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.78300000000000003}]},{"osd":5,"last update":"Tue Sep 17 00:17:06 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.42399999999999999,"15min":0.41999999999999998},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.63100000000000001,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.442,"15min":0.437},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.63200000000000001}]},{"osd":6,"last update":"Tue Sep 17 00:16:28 2024","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.627,"5min":0.627,"15min":0.627},"last":0.56999999999999995}]},{"osd":7,"last update":"Tue Sep 17 00:16:50 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.66400000000000003}]}]},{"osd":5,"up_from":39,"seq":167503724573,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5704,"kb_used_data":576,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739448,"statfs":{"total":95995035648,"available":95989194752,"internally_reserved":0,"allocated":589824,"data_stored":240660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 00:17:04 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.51300000000000001,"15min":0.51200000000000001},"min":{"1min":0.39600000000000002,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.76900000000000002,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.53100000000000003,"15min":0.52600000000000002},"min":{"1min":0.38500000000000001,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.54700000000000004}]},{"osd":1,"last update":"Tue Sep 17 00:17:04 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.51800000000000002,"15min":0.51300000000000001},"min":{"1min":0.36299999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.53600000000000003,"15min":0.53400000000000003},"min":{"1min":0.38100000000000001,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.57099999999999995}]},{"osd":2,"last update":"Tue Sep 17 00:17:04 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.52100000000000002,"15min":0.51600000000000001},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.626},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.54600000000000004,"15min":0.54300000000000004},"min":{"1min":0.40100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.504}]},{"osd":3,"last update":"Tue Sep 17 00:17:04 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.54200000000000004,"15min":0.53600000000000003},"min":{"1min":0.40699999999999997,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.54900000000000004,"15min":0.54300000000000004},"min":{"1min":0.35399999999999998,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.79300000000000004,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.59999999999999998}]},{"osd":4,"last update":"Tue Sep 17 00:17:04 2024","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.39400000000000002,"15min":0.38500000000000001},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.39700000000000002,"15min":0.38400000000000001},"min":{"1min":0.30599999999999999,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"last":0.52800000000000002}]},{"osd":6,"last update":"Tue Sep 17 00:16:29 2024","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"last":0.47999999999999998}]},{"osd":7,"last update":"Tue Sep 17 00:16:52 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.61499999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T00:17:20.719 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 3aad27a2-7489-11ef-bceb-c7b262605968 -- ceph pg dump --format=json 2024-09-17T00:17:20.719 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:17:20 smithi099 ceph-mon[24594]: from='client.24496 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T00:17:20.936 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T00:17:21.716 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:17:21 smithi099 ceph-mon[24594]: pgmap v38: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:17:21.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:17:21 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:17:21.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:17:21 smithi002 ceph-mon[28203]: pgmap v38: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:17:21.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:17:21 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:17:21.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:17:21 smithi002 ceph-mon[23737]: pgmap v38: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:17:21.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:17:21 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:17:22.823 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-17T00:17:22.824 INFO:teuthology.orchestra.run.smithi002.stderr:dumped all 2024-09-17T00:17:23.431 INFO:teuthology.orchestra.run.smithi002.stdout:{"pg_ready":true,"pg_map":{"version":39,"stamp":"2024-09-17T00:17:22.359640+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":749961216,"kb_used":48968,"kb_used_data":6152,"kb_used_omap":0,"kb_used_meta":42752,"kb_avail":749912248,"statfs":{"total":767960285184,"available":767910141952,"internally_reserved":0,"allocated":6299648,"data_stored":3510946,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":43778048},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002359"},"pg_stats":[{"pgid":"1.0","version":"60'128","reported_seq":267,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-17T00:16:10.433169+0000","last_change":"2024-09-17T00:15:53.556149+0000","last_active":"2024-09-17T00:16:10.433169+0000","last_peered":"2024-09-17T00:16:10.433169+0000","last_clean":"2024-09-17T00:16:10.433169+0000","last_became_active":"2024-09-17T00:15:53.555687+0000","last_became_peered":"2024-09-17T00:15:53.555687+0000","last_unstale":"2024-09-17T00:16:10.433169+0000","last_undegraded":"2024-09-17T00:16:10.433169+0000","last_fullsized":"2024-09-17T00:16:10.433169+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-17T00:13:54.787965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-17T00:13:54.787965+0000","last_clean_scrub_stamp":"2024-09-17T00:13:54.787965+0000","objects_scrubbed":0,"log_size":128,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-18T01:30:48.289980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":55,"seq":236223201299,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6096,"kb_used_data":1096,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":93739056,"statfs":{"total":95995035648,"available":95988793344,"internally_reserved":0,"allocated":1122304,"data_stored":769588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 00:16:50 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.501,"15min":0.501},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.28899999999999998}]},{"osd":1,"last update":"Tue Sep 17 00:16:50 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.33800000000000002}]},{"osd":2,"last update":"Tue Sep 17 00:16:50 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.41499999999999998}]},{"osd":3,"last update":"Tue Sep 17 00:16:50 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.871,"5min":0.871,"15min":0.871},"last":0.495},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.47899999999999998}]},{"osd":4,"last update":"Tue Sep 17 00:16:50 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.46100000000000002}]},{"osd":5,"last update":"Tue Sep 17 00:16:50 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.439},{"interface":"front","average":{"1min":0.504,"5min":0.504,"15min":0.504},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.48399999999999999}]},{"osd":6,"last update":"Tue Sep 17 00:16:50 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.53900000000000003}]}]},{"osd":6,"up_from":47,"seq":201863462938,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6152,"kb_used_data":1088,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93739000,"statfs":{"total":95995035648,"available":95988736000,"internally_reserved":0,"allocated":1114112,"data_stored":769000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 00:16:30 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.60799999999999998}]},{"osd":1,"last update":"Tue Sep 17 00:16:30 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.627}]},{"osd":2,"last update":"Tue Sep 17 00:16:30 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.55600000000000005}]},{"osd":3,"last update":"Tue Sep 17 00:16:30 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.57799999999999996}]},{"osd":4,"last update":"Tue Sep 17 00:16:30 2024","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"last":0.40699999999999997}]},{"osd":5,"last update":"Tue Sep 17 00:16:30 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.47599999999999998}]},{"osd":7,"last update":"Tue Sep 17 00:16:50 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.49399999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574896,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6344,"kb_used_data":576,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738808,"statfs":{"total":95995035648,"available":95988539392,"internally_reserved":0,"allocated":589824,"data_stored":240660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 00:16:35 2024","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.34599999999999997,"15min":0.32600000000000001},"min":{"1min":0.247,"5min":0.214,"15min":0.214},"max":{"1min":0.59099999999999997,"5min":0.63600000000000001,"15min":0.63600000000000001},"last":0.436},{"interface":"front","average":{"1min":0.36599999999999999,"5min":0.36199999999999999,"15min":0.34399999999999997},"min":{"1min":0.26400000000000001,"5min":0.214,"15min":0.214},"max":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"last":0.41599999999999998}]},{"osd":2,"last update":"Tue Sep 17 00:16:58 2024","interfaces":[{"interface":"back","average":{"1min":0.38,"5min":0.374,"15min":0.36399999999999999},"min":{"1min":0.26800000000000002,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":1.01,"5min":1.01,"15min":1.01},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.40699999999999997,"15min":0.39400000000000002},"min":{"1min":0.30499999999999999,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.505}]},{"osd":3,"last update":"Tue Sep 17 00:16:22 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.48899999999999999,"15min":0.49099999999999999},"min":{"1min":0.35999999999999999,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.371}]},{"osd":4,"last update":"Tue Sep 17 00:16:41 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.60599999999999998,"15min":0.61399999999999999},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.93100000000000005,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.61199999999999999,"15min":0.62},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.44900000000000001}]},{"osd":5,"last update":"Tue Sep 17 00:17:06 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.59699999999999998,"15min":0.60199999999999998},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.40000000000000002},{"interface":"front","average":{"1min":1.0960000000000001,"5min":0.73599999999999999,"15min":0.67600000000000005},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":11.391999999999999,"5min":11.391999999999999,"15min":11.391999999999999},"last":0.66500000000000004}]},{"osd":6,"last update":"Tue Sep 17 00:16:26 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.48299999999999998}]},{"osd":7,"last update":"Tue Sep 17 00:16:52 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.044,"5min":1.044,"15min":1.044},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.49399999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738422,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6928,"kb_used_data":1096,"kb_used_omap":0,"kb_used_meta":5824,"kb_avail":93738224,"statfs":{"total":95995035648,"available":95987941376,"internally_reserved":0,"allocated":1122304,"data_stored":769588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5963776},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Sep 17 00:16:39 2024","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.36199999999999999,"15min":0.34499999999999997},"min":{"1min":0.27000000000000002,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.41999999999999998,"5min":0.38800000000000001,"15min":0.35999999999999999},"min":{"1min":0.30599999999999999,"5min":0.246,"15min":0.246},"max":{"1min":0.61599999999999999,"5min":0.65400000000000003,"15min":0.65400000000000003},"last":0.253}]},{"osd":2,"last update":"Tue Sep 17 00:17:01 2024","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.40200000000000002,"15min":0.39900000000000002},"min":{"1min":0.29099999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.47899999999999998,"5min":0.64300000000000002,"15min":0.64300000000000002},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.41299999999999998,"5min":0.43099999999999999,"15min":0.42499999999999999},"min":{"1min":0.32800000000000001,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.51900000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"last":0.311}]},{"osd":3,"last update":"Tue Sep 17 00:16:21 2024","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.45400000000000001,"15min":0.45600000000000002},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.58299999999999996,"5min":0.626,"15min":0.626},"last":0.316},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.503,"15min":0.50600000000000001},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.371}]},{"osd":4,"last update":"Tue Sep 17 00:16:42 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.63,"15min":0.63200000000000001},"min":{"1min":0.45500000000000002,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.629,"5min":0.64100000000000001,"15min":0.64300000000000002},"min":{"1min":0.48299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.85299999999999998,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.65400000000000003}]},{"osd":5,"last update":"Tue Sep 17 00:17:06 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.59199999999999997,"15min":0.58799999999999997},"min":{"1min":0.438,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.93100000000000005,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.61699999999999999,"15min":0.621},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.59199999999999997}]},{"osd":6,"last update":"Tue Sep 17 00:16:27 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.623,"15min":0.623},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":1.0189999999999999,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.624,"5min":0.624,"15min":0.624},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.45400000000000001}]},{"osd":7,"last update":"Tue Sep 17 00:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.628},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.98299999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":0.60699999999999998}]}]},{"osd":2,"up_from":18,"seq":77309411372,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6276,"kb_used_data":572,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":93738876,"statfs":{"total":95995035648,"available":95988609024,"internally_reserved":0,"allocated":585728,"data_stored":240395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 00:16:54 2024","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.37,"15min":0.35799999999999998},"min":{"1min":0.25900000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.55600000000000005,"5min":0.65100000000000002,"15min":0.65100000000000002},"last":0.28399999999999997},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.39000000000000001,"15min":0.38300000000000001},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"last":0.36199999999999999}]},{"osd":1,"last update":"Tue Sep 17 00:16:54 2024","interfaces":[{"interface":"back","average":{"1min":0.39100000000000001,"5min":0.40100000000000002,"15min":0.39300000000000002},"min":{"1min":0.28399999999999997,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.67900000000000005,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.33100000000000002},{"interface":"front","average":{"1min":0.41799999999999998,"5min":0.41699999999999998,"15min":0.40699999999999997},"min":{"1min":0.28199999999999997,"5min":0.25,"15min":0.25},"max":{"1min":0.58199999999999996,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.26800000000000002}]},{"osd":3,"last update":"Tue Sep 17 00:16:23 2024","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.47999999999999998,"15min":0.48099999999999998},"min":{"1min":0.36299999999999999,"5min":0.307,"15min":0.307},"max":{"1min":0.59999999999999998,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.505,"15min":0.50900000000000001},"min":{"1min":0.317,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.629,"5min":0.67700000000000005,"15min":0.67700000000000005},"last":0.32200000000000001}]},{"osd":4,"last update":"Tue Sep 17 00:16:42 2024","interfaces":[{"interface":"back","average":{"1min":0.81699999999999995,"5min":0.625,"15min":0.59299999999999997},"min":{"1min":0.40899999999999997,"5min":0.373,"15min":0.373},"max":{"1min":7.2960000000000003,"5min":7.2960000000000003,"15min":7.2960000000000003},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.60799999999999998,"15min":0.60699999999999998},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.82899999999999996,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.47099999999999997}]},{"osd":5,"last update":"Tue Sep 17 00:17:03 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.59399999999999997,"15min":0.59999999999999998},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.77100000000000002,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.63300000000000001,"15min":0.64000000000000001},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.84999999999999998,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.44600000000000001}]},{"osd":6,"last update":"Tue Sep 17 00:16:27 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.43099999999999999}]},{"osd":7,"last update":"Tue Sep 17 00:16:50 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.54500000000000004}]}]},{"osd":3,"up_from":26,"seq":111669149735,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5700,"kb_used_data":572,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739452,"statfs":{"total":95995035648,"available":95989198848,"internally_reserved":0,"allocated":585728,"data_stored":240395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 00:17:21 2024","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.39600000000000002,"15min":0.40100000000000002},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.51800000000000002,"5min":0.57799999999999996,"15min":0.57799999999999996},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.35799999999999998,"5min":0.41999999999999998,"15min":0.42599999999999999},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.46999999999999997,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.45000000000000001}]},{"osd":1,"last update":"Tue Sep 17 00:17:21 2024","interfaces":[{"interface":"back","average":{"1min":0.379,"5min":0.41899999999999998,"15min":0.42699999999999999},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.53900000000000003,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.375,"5min":0.435,"15min":0.442},"min":{"1min":0.28999999999999998,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.51400000000000001,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.38600000000000001}]},{"osd":2,"last update":"Tue Sep 17 00:17:21 2024","interfaces":[{"interface":"back","average":{"1min":0.39100000000000001,"5min":0.46300000000000002,"15min":0.47899999999999998},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.505,"5min":0.64500000000000002,"15min":0.64500000000000002},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.40300000000000002,"5min":0.44500000000000001,"15min":0.45300000000000001},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.56899999999999995,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.45800000000000002}]},{"osd":4,"last update":"Tue Sep 17 00:16:43 2024","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.60199999999999998,"15min":0.59599999999999997},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.5},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.58599999999999997,"15min":0.58399999999999996},"min":{"1min":0.45400000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.51300000000000001}]},{"osd":5,"last update":"Tue Sep 17 00:17:06 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.60299999999999998,"15min":0.60599999999999998},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.72899999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.622,"15min":0.626},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.82899999999999996,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.54400000000000004}]},{"osd":6,"last update":"Tue Sep 17 00:16:28 2024","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.873,"5min":0.873,"15min":0.873},"last":0.47499999999999998}]},{"osd":7,"last update":"Tue Sep 17 00:16:51 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.628,"5min":0.628,"15min":0.628},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.60399999999999998}]}]},{"osd":4,"up_from":31,"seq":133143986210,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5768,"kb_used_data":576,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":93739384,"statfs":{"total":95995035648,"available":95989129216,"internally_reserved":0,"allocated":589824,"data_stored":240660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 00:16:41 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.55700000000000005,"15min":0.55100000000000005},"min":{"1min":0.41899999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.873,"5min":0.873,"15min":0.873},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.57799999999999996,"15min":0.57699999999999996},"min":{"1min":0.35299999999999998,"5min":0.318,"15min":0.318},"max":{"1min":0.83299999999999996,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.54100000000000004}]},{"osd":1,"last update":"Tue Sep 17 00:16:41 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.56200000000000006,"15min":0.56000000000000005},"min":{"1min":0.38700000000000001,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.57499999999999996,"15min":0.56899999999999995},"min":{"1min":0.46100000000000002,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.52200000000000002}]},{"osd":2,"last update":"Tue Sep 17 00:16:41 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.57699999999999996,"15min":0.57299999999999995},"min":{"1min":0.40100000000000002,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.85399999999999998,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.57099999999999995,"15min":0.56799999999999995},"min":{"1min":0.38500000000000001,"5min":0.374,"15min":0.374},"max":{"1min":0.79500000000000004,"5min":0.88,"15min":0.88},"last":0.44500000000000001}]},{"osd":3,"last update":"Tue Sep 17 00:16:41 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.41099999999999998,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.56100000000000005,"15min":0.55500000000000005},"min":{"1min":0.434,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.53000000000000003}]},{"osd":5,"last update":"Tue Sep 17 00:17:06 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.42399999999999999,"15min":0.41999999999999998},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.63100000000000001,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.38},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.442,"15min":0.437},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.50900000000000001}]},{"osd":6,"last update":"Tue Sep 17 00:16:28 2024","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.627,"5min":0.627,"15min":0.627},"last":0.55600000000000005}]},{"osd":7,"last update":"Tue Sep 17 00:16:50 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.498},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.49199999999999999}]}]},{"osd":5,"up_from":39,"seq":167503724574,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5704,"kb_used_data":576,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739448,"statfs":{"total":95995035648,"available":95989194752,"internally_reserved":0,"allocated":589824,"data_stored":240660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 17 00:17:04 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.51300000000000001,"15min":0.51200000000000001},"min":{"1min":0.39600000000000002,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.76900000000000002,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.53100000000000003,"15min":0.52600000000000002},"min":{"1min":0.38500000000000001,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.437}]},{"osd":1,"last update":"Tue Sep 17 00:17:04 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.51800000000000002,"15min":0.51300000000000001},"min":{"1min":0.36299999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.53600000000000003,"15min":0.53400000000000003},"min":{"1min":0.38100000000000001,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.46800000000000003}]},{"osd":2,"last update":"Tue Sep 17 00:17:04 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.52100000000000002,"15min":0.51600000000000001},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.54600000000000004,"15min":0.54300000000000004},"min":{"1min":0.40100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.39800000000000002}]},{"osd":3,"last update":"Tue Sep 17 00:17:04 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.54200000000000004,"15min":0.53600000000000003},"min":{"1min":0.40699999999999997,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.374},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.54900000000000004,"15min":0.54300000000000004},"min":{"1min":0.35399999999999998,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.79300000000000004,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.36099999999999999}]},{"osd":4,"last update":"Tue Sep 17 00:17:04 2024","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.39400000000000002,"15min":0.38500000000000001},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.39700000000000002,"15min":0.38400000000000001},"min":{"1min":0.30599999999999999,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"last":0.40999999999999998}]},{"osd":6,"last update":"Tue Sep 17 00:16:29 2024","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"last":0.504}]},{"osd":7,"last update":"Tue Sep 17 00:16:52 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.49099999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-17T00:17:23.433 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-09-17T00:17:23.433 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-17T00:17:23.433 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-09-17T00:17:23.433 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 3aad27a2-7489-11ef-bceb-c7b262605968 -- ceph health --format=json 2024-09-17T00:17:23.651 INFO:teuthology.orchestra.run.smithi002.stderr:Inferring config /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/mon.a/config 2024-09-17T00:17:23.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:17:23 smithi002 ceph-mon[23737]: pgmap v39: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:17:23.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:17:23 smithi002 ceph-mon[23737]: from='client.24502 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T00:17:23.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:17:23 smithi002 ceph-mon[28203]: pgmap v39: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:17:23.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:17:23 smithi002 ceph-mon[28203]: from='client.24502 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T00:17:23.966 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:17:23 smithi099 ceph-mon[24594]: pgmap v39: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:17:23.966 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:17:23 smithi099 ceph-mon[24594]: from='client.24502 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-17T00:17:25.216 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:17:24 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[25772]: ::ffff:172.21.15.99 - - [17/Sep/2024:00:17:24] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-09-17T00:17:25.407 INFO:teuthology.orchestra.run.smithi002.stdout: 2024-09-17T00:17:25.407 INFO:teuthology.orchestra.run.smithi002.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-17T00:17:25.906 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:17:25 smithi099 ceph-mon[24594]: pgmap v40: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:17:25.906 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:17:25 smithi099 ceph-mon[24594]: from='client.? 172.21.15.2:0/2447281762' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-17T00:17:25.973 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:17:25 smithi002 ceph-mon[28203]: pgmap v40: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:17:25.974 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:17:25 smithi002 ceph-mon[28203]: from='client.? 172.21.15.2:0/2447281762' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-17T00:17:25.974 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:17:25 smithi002 ceph-mon[23737]: pgmap v40: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:17:25.974 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:17:25 smithi002 ceph-mon[23737]: from='client.? 172.21.15.2:0/2447281762' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-17T00:17:25.996 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-09-17T00:17:25.996 INFO:tasks.cephadm:Setup complete, yielding 2024-09-17T00:17:25.996 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-09-17T00:17:26.006 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi002.front.sepia.ceph.com 2024-09-17T00:17:26.007 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3aad27a2-7489-11ef-bceb-c7b262605968 -- bash -c 'ceph config set mgr mgr/cephadm/use_repo_digest false --force' 2024-09-17T00:17:27.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:17:27 smithi099 ceph-mon[24594]: pgmap v41: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:17:27.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:17:27 smithi002 ceph-mon[28203]: pgmap v41: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:17:27.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:17:27 smithi002 ceph-mon[23737]: pgmap v41: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:17:28.320 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Sep 17 00:17:28 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-alertmanager-a[46262]: level=info ts=2024-09-17T00:17:28.127Z caller=cluster.go:688 component=cluster msg="gossip settled; proceeding" elapsed=10.003953609s 2024-09-17T00:17:28.577 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-09-17T00:17:28.586 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi002.front.sepia.ceph.com 2024-09-17T00:17:28.586 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3aad27a2-7489-11ef-bceb-c7b262605968 -e sha1=c2ddf9721ef254645201cdbabd9c8f49c30a586f -- bash -c 'radosgw-admin realm create --rgw-realm=r --default' 2024-09-17T00:17:28.957 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:28 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:28+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_provisioning v2" 2024-09-17T00:17:29.189 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:17:29 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: ::ffff:172.21.15.99 - - [17/Sep/2024:00:17:29] "GET /metrics HTTP/1.1" 200 191469 "" "Prometheus/2.33.4" 2024-09-17T00:17:29.189 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:17:28 smithi002 ceph-mon[28203]: from='client.? ' entity='client.admin' 2024-09-17T00:17:29.189 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:17:28 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:17:29.189 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:17:28 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:17:29.189 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:17:28 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:17:29.189 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:17:28 smithi002 ceph-mon[23737]: from='client.? ' entity='client.admin' 2024-09-17T00:17:29.190 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:17:28 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:17:29.190 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:17:28 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:17:29.190 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:17:28 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:17:29.209 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:17:28 smithi099 ceph-mon[24594]: from='client.? ' entity='client.admin' 2024-09-17T00:17:29.209 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:17:28 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:17:29.209 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:17:28 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:17:29.209 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:17:28 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:17:29.209 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:28 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:28+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2024-09-17T00:17:29.209 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:28 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:28+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2024-09-17T00:17:29.210 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:28 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:28+0000 lvl=info msg="Executing migration" logger=migrator id="copy dashboard_provisioning v1 to v2" 2024-09-17T00:17:29.210 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:28 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:28+0000 lvl=info msg="Executing migration" logger=migrator id="drop dashboard_provisioning_tmp_qwerty" 2024-09-17T00:17:29.210 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:28 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:28+0000 lvl=info msg="Executing migration" logger=migrator id="Add check_sum column" 2024-09-17T00:17:29.210 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:28 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:28+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for dashboard_title" 2024-09-17T00:17:29.210 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:28 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:28+0000 lvl=info msg="Executing migration" logger=migrator id="delete tags for deleted dashboards" 2024-09-17T00:17:29.210 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:28 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:28+0000 lvl=info msg="Executing migration" logger=migrator id="delete stars for deleted dashboards" 2024-09-17T00:17:29.210 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:28 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:28+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for dashboard_is_folder" 2024-09-17T00:17:29.210 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:28 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:28+0000 lvl=info msg="Executing migration" logger=migrator id="create data_source table" 2024-09-17T00:17:29.210 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:28 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:28+0000 lvl=info msg="Executing migration" logger=migrator id="add index data_source.account_id" 2024-09-17T00:17:29.210 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:28 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:28+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index data_source.account_id_name" 2024-09-17T00:17:29.210 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:28 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:28+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_data_source_account_id - v1" 2024-09-17T00:17:29.210 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:28 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:28+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_data_source_account_id_name - v1" 2024-09-17T00:17:29.210 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table data_source to data_source_v1 - v1" 2024-09-17T00:17:29.211 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="create data_source table v2" 2024-09-17T00:17:29.211 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_data_source_org_id - v2" 2024-09-17T00:17:29.211 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_data_source_org_id_name - v2" 2024-09-17T00:17:29.211 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="copy data_source v1 to v2" 2024-09-17T00:17:29.211 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table data_source_v1 #2" 2024-09-17T00:17:29.211 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="Add column with_credentials" 2024-09-17T00:17:29.211 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="Add secure json data column" 2024-09-17T00:17:29.211 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="Update data_source table charset" 2024-09-17T00:17:29.211 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="Update initial version to 1" 2024-09-17T00:17:29.211 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="Add read_only data column" 2024-09-17T00:17:29.211 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="Migrate logging ds to loki ds" 2024-09-17T00:17:29.211 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="Update json_data with nulls" 2024-09-17T00:17:29.211 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="Add uid column" 2024-09-17T00:17:29.211 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="Update uid value" 2024-09-17T00:17:29.212 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index datasource_org_id_uid" 2024-09-17T00:17:29.212 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index datasource_org_id_is_default" 2024-09-17T00:17:29.212 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="create api_key table" 2024-09-17T00:17:29.212 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.account_id" 2024-09-17T00:17:29.212 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.key" 2024-09-17T00:17:29.212 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.account_id_name" 2024-09-17T00:17:29.212 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_api_key_account_id - v1" 2024-09-17T00:17:29.212 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_api_key_key - v1" 2024-09-17T00:17:29.212 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_api_key_account_id_name - v1" 2024-09-17T00:17:29.212 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table api_key to api_key_v1 - v1" 2024-09-17T00:17:29.212 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="create api_key table v2" 2024-09-17T00:17:29.213 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_api_key_org_id - v2" 2024-09-17T00:17:29.214 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_api_key_key - v2" 2024-09-17T00:17:29.214 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_api_key_org_id_name - v2" 2024-09-17T00:17:29.214 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="copy api_key v1 to v2" 2024-09-17T00:17:29.214 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table api_key_v1" 2024-09-17T00:17:29.214 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="Update api_key table charset" 2024-09-17T00:17:29.214 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="Add expires to api_key table" 2024-09-17T00:17:29.214 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="Add service account foreign key" 2024-09-17T00:17:29.214 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_snapshot table v4" 2024-09-17T00:17:29.214 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="drop table dashboard_snapshot_v4 #1" 2024-09-17T00:17:29.214 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_snapshot table v5 #2" 2024-09-17T00:17:29.214 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_snapshot_key - v5" 2024-09-17T00:17:29.214 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_snapshot_delete_key - v5" 2024-09-17T00:17:29.214 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_snapshot_user_id - v5" 2024-09-17T00:17:29.215 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard_snapshot to mediumtext v2" 2024-09-17T00:17:29.215 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard_snapshot table charset" 2024-09-17T00:17:29.215 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="Add column external_delete_url to dashboard_snapshots table" 2024-09-17T00:17:29.215 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="Add encrypted dashboard json column" 2024-09-17T00:17:29.215 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="Change dashboard_encrypted column to MEDIUMBLOB" 2024-09-17T00:17:29.215 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="create quota table v1" 2024-09-17T00:17:29.215 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_quota_org_id_user_id_target - v1" 2024-09-17T00:17:29.215 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="Update quota table charset" 2024-09-17T00:17:29.215 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="create plugin_setting table" 2024-09-17T00:17:29.215 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2024-09-17T00:17:29.215 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="Add column plugin_version to plugin_settings" 2024-09-17T00:17:29.216 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="Update plugin_setting table charset" 2024-09-17T00:17:29.216 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="create session table" 2024-09-17T00:17:29.216 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table playlist table" 2024-09-17T00:17:29.216 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table playlist_item table" 2024-09-17T00:17:29.216 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="create playlist table v2" 2024-09-17T00:17:29.216 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="create playlist item table v2" 2024-09-17T00:17:29.216 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="Update playlist table charset" 2024-09-17T00:17:29.216 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="Update playlist_item table charset" 2024-09-17T00:17:29.216 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="drop preferences table v2" 2024-09-17T00:17:29.216 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="drop preferences table v3" 2024-09-17T00:17:29.216 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="create preferences table v3" 2024-09-17T00:17:29.216 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="Update preferences table charset" 2024-09-17T00:17:29.216 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="Add column team_id in preferences" 2024-09-17T00:17:29.217 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="Update team_id column values in preferences" 2024-09-17T00:17:29.217 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="Add column week_start in preferences" 2024-09-17T00:17:29.217 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="create alert table v1" 2024-09-17T00:17:29.217 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert org_id & id " 2024-09-17T00:17:29.217 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert state" 2024-09-17T00:17:29.217 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert dashboard_id" 2024-09-17T00:17:29.217 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="Create alert_rule_tag table v1" 2024-09-17T00:17:29.217 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index alert_rule_tag.alert_id_tag_id" 2024-09-17T00:17:29.217 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" 2024-09-17T00:17:29.217 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" 2024-09-17T00:17:29.217 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="Create alert_rule_tag table v2" 2024-09-17T00:17:29.217 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" 2024-09-17T00:17:29.217 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="copy alert_rule_tag v1 to v2" 2024-09-17T00:17:29.218 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="drop table alert_rule_tag_v1" 2024-09-17T00:17:29.218 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_notification table v1" 2024-09-17T00:17:29.218 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="Add column is_default" 2024-09-17T00:17:29.218 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="Add column frequency" 2024-09-17T00:17:29.218 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="Add column send_reminder" 2024-09-17T00:17:29.218 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="Add column disable_resolve_message" 2024-09-17T00:17:29.218 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert_notification org_id & name" 2024-09-17T00:17:29.218 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert table charset" 2024-09-17T00:17:29.218 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert_notification table charset" 2024-09-17T00:17:29.218 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="create notification_journal table v1" 2024-09-17T00:17:29.218 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="add index notification_journal org_id & alert_id & notifier_id" 2024-09-17T00:17:29.218 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="drop alert_notification_journal" 2024-09-17T00:17:29.218 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_notification_state table v1" 2024-09-17T00:17:29.218 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert_notification_state org_id & alert_id & notifier_id" 2024-09-17T00:17:29.219 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="Add for to alert table" 2024-09-17T00:17:29.219 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="Add column uid in alert_notification" 2024-09-17T00:17:29.219 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="Update uid column values in alert_notification" 2024-09-17T00:17:29.219 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index alert_notification_org_id_uid" 2024-09-17T00:17:29.219 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="Remove unique index org_id_name" 2024-09-17T00:17:29.219 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="Add column secure_settings in alert_notification" 2024-09-17T00:17:29.219 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert.settings to mediumtext" 2024-09-17T00:17:29.219 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="Add non-unique index alert_notification_state_alert_id" 2024-09-17T00:17:29.219 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="Add non-unique index alert_rule_tag_alert_id" 2024-09-17T00:17:29.219 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old annotation table v4" 2024-09-17T00:17:29.219 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="create annotation table v5" 2024-09-17T00:17:29.219 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 0 v3" 2024-09-17T00:17:29.219 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 1 v3" 2024-09-17T00:17:29.219 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 2 v3" 2024-09-17T00:17:29.219 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 3 v3" 2024-09-17T00:17:29.220 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 4 v3" 2024-09-17T00:17:29.220 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="Update annotation table charset" 2024-09-17T00:17:29.220 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="Add column region_id to annotation table" 2024-09-17T00:17:29.220 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="Drop category_id index" 2024-09-17T00:17:29.220 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="Add column tags to annotation table" 2024-09-17T00:17:29.220 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="Create annotation_tag table v2" 2024-09-17T00:17:29.220 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index annotation_tag.annotation_id_tag_id" 2024-09-17T00:17:29.220 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" 2024-09-17T00:17:29.220 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table annotation_tag to annotation_tag_v2 - v2" 2024-09-17T00:17:29.220 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="Create annotation_tag table v3" 2024-09-17T00:17:29.220 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" 2024-09-17T00:17:29.220 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="copy annotation_tag v2 to v3" 2024-09-17T00:17:29.220 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="drop table annotation_tag_v2" 2024-09-17T00:17:29.220 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert annotations and set TEXT to empty" 2024-09-17T00:17:29.220 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="Add created time to annotation table" 2024-09-17T00:17:29.221 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="Add updated time to annotation table" 2024-09-17T00:17:29.221 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for created in annotation table" 2024-09-17T00:17:29.221 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for updated in annotation table" 2024-09-17T00:17:29.221 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="Convert existing annotations from seconds to milliseconds" 2024-09-17T00:17:29.221 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="Add epoch_end column" 2024-09-17T00:17:29.221 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for epoch_end" 2024-09-17T00:17:29.221 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="Make epoch_end the same as epoch" 2024-09-17T00:17:29.221 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="Move region to single row" 2024-09-17T00:17:29.221 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_epoch from annotation table" 2024-09-17T00:17:29.221 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2024-09-17T00:17:29.221 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2024-09-17T00:17:29.221 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for org_id_epoch_end_epoch on annotation table" 2024-09-17T00:17:29.221 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_epoch_epoch_end from annotation table" 2024-09-17T00:17:29.221 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for alert_id on annotation table" 2024-09-17T00:17:29.222 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="create test_data table" 2024-09-17T00:17:29.222 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_version table v1" 2024-09-17T00:17:29.222 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_version.dashboard_id" 2024-09-17T00:17:29.222 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2024-09-17T00:17:29.222 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="Set dashboard version to 1 where 0" 2024-09-17T00:17:29.222 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="save existing dashboard data in dashboard_version table v1" 2024-09-17T00:17:29.222 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard_version.data to mediumtext v1" 2024-09-17T00:17:29.222 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="create team table" 2024-09-17T00:17:29.222 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="add index team.org_id" 2024-09-17T00:17:29.222 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_org_id_name" 2024-09-17T00:17:29.222 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="create team member table" 2024-09-17T00:17:29.222 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_member.org_id" 2024-09-17T00:17:29.222 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_member_org_id_team_id_user_id" 2024-09-17T00:17:29.223 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_member.team_id" 2024-09-17T00:17:29.223 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="Add column email to team table" 2024-09-17T00:17:29.223 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="Add column external to team_member table" 2024-09-17T00:17:29.223 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="Add column permission to team_member table" 2024-09-17T00:17:29.223 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard acl table" 2024-09-17T00:17:29.223 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_dashboard_id" 2024-09-17T00:17:29.223 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_acl_dashboard_id_user_id" 2024-09-17T00:17:29.223 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_acl_dashboard_id_team_id" 2024-09-17T00:17:29.223 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_user_id" 2024-09-17T00:17:29.223 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_team_id" 2024-09-17T00:17:29.223 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_org_id_role" 2024-09-17T00:17:29.223 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_permission" 2024-09-17T00:17:29.223 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="save default acl rules in dashboard_acl table" 2024-09-17T00:17:29.224 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="delete acl rules for deleted dashboards and folders" 2024-09-17T00:17:29.224 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="create tag table" 2024-09-17T00:17:29.224 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="add index tag.key_value" 2024-09-17T00:17:29.224 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="create login attempt table" 2024-09-17T00:17:29.224 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="add index login_attempt.username" 2024-09-17T00:17:29.224 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_login_attempt_username - v1" 2024-09-17T00:17:29.224 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2024-09-17T00:17:29.224 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="create login_attempt v2" 2024-09-17T00:17:29.224 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_login_attempt_username - v2" 2024-09-17T00:17:29.224 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="copy login_attempt v1 to v2" 2024-09-17T00:17:29.224 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="drop login_attempt_tmp_qwerty" 2024-09-17T00:17:29.224 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="create user auth table" 2024-09-17T00:17:29.224 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_user_auth_auth_module_auth_id - v1" 2024-09-17T00:17:29.224 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="alter user_auth.auth_id to length 190" 2024-09-17T00:17:29.225 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth access token to user_auth" 2024-09-17T00:17:29.225 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth refresh token to user_auth" 2024-09-17T00:17:29.225 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth token type to user_auth" 2024-09-17T00:17:29.225 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth expiry to user_auth" 2024-09-17T00:17:29.225 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="Add index to user_id column in user_auth" 2024-09-17T00:17:29.465 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="create server_lock table" 2024-09-17T00:17:29.466 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="add index server_lock.operation_uid" 2024-09-17T00:17:29.466 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="create user auth token table" 2024-09-17T00:17:29.466 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_auth_token.auth_token" 2024-09-17T00:17:29.466 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_auth_token.prev_auth_token" 2024-09-17T00:17:29.466 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="add index user_auth_token.user_id" 2024-09-17T00:17:29.466 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="Add revoked_at to the user auth token" 2024-09-17T00:17:29.466 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="create cache_data table" 2024-09-17T00:17:29.466 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index cache_data.cache_key" 2024-09-17T00:17:29.466 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="create short_url table v1" 2024-09-17T00:17:29.466 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="add index short_url.org_id-uid" 2024-09-17T00:17:29.466 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="delete alert_definition table" 2024-09-17T00:17:29.466 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="recreate alert_definition table" 2024-09-17T00:17:29.466 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition on org_id and title columns" 2024-09-17T00:17:29.467 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition on org_id and uid columns" 2024-09-17T00:17:29.467 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_definition table data column to mediumtext in mysql" 2024-09-17T00:17:29.467 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="drop index in alert_definition on org_id and title columns" 2024-09-17T00:17:29.467 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="drop index in alert_definition on org_id and uid columns" 2024-09-17T00:17:29.467 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index in alert_definition on org_id and title columns" 2024-09-17T00:17:29.467 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index in alert_definition on org_id and uid columns" 2024-09-17T00:17:29.467 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="Add column paused in alert_definition" 2024-09-17T00:17:29.467 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="drop alert_definition table" 2024-09-17T00:17:29.467 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="delete alert_definition_version table" 2024-09-17T00:17:29.467 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="recreate alert_definition_version table" 2024-09-17T00:17:29.467 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition_version table on alert_definition_id and version columns" 2024-09-17T00:17:29.467 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition_version table on alert_definition_uid and version columns" 2024-09-17T00:17:29.467 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_definition_version table data column to mediumtext in mysql" 2024-09-17T00:17:29.467 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="drop alert_definition_version table" 2024-09-17T00:17:29.468 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_instance table" 2024-09-17T00:17:29.468 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_instance table on def_org_id, def_uid and current_state columns" 2024-09-17T00:17:29.468 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_instance table on def_org_id, current_state columns" 2024-09-17T00:17:29.468 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="add column current_state_end to alert_instance" 2024-09-17T00:17:29.468 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="remove index def_org_id, def_uid, current_state on alert_instance" 2024-09-17T00:17:29.468 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="remove index def_org_id, current_state on alert_instance" 2024-09-17T00:17:29.468 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="rename def_org_id to rule_org_id in alert_instance" 2024-09-17T00:17:29.468 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="rename def_uid to rule_uid in alert_instance" 2024-09-17T00:17:29.468 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="add index rule_org_id, rule_uid, current_state on alert_instance" 2024-09-17T00:17:29.469 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="add index rule_org_id, current_state on alert_instance" 2024-09-17T00:17:29.469 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_rule table" 2024-09-17T00:17:29.469 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id and title columns" 2024-09-17T00:17:29.469 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id and uid columns" 2024-09-17T00:17:29.469 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id, namespace_uid, group_uid columns" 2024-09-17T00:17:29.469 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_rule table data column to mediumtext in mysql" 2024-09-17T00:17:29.469 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="add column for to alert_rule" 2024-09-17T00:17:29.469 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="add column annotations to alert_rule" 2024-09-17T00:17:29.470 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="add column labels to alert_rule" 2024-09-17T00:17:29.470 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="remove unique index from alert_rule on org_id, title columns" 2024-09-17T00:17:29.470 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id, namespase_uid and title columns" 2024-09-17T00:17:29.470 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="add dashboard_uid column to alert_rule" 2024-09-17T00:17:29.470 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="add panel_id column to alert_rule" 2024-09-17T00:17:29.470 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" 2024-09-17T00:17:29.470 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_rule_version table" 2024-09-17T00:17:29.470 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" 2024-09-17T00:17:29.470 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" 2024-09-17T00:17:29.470 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_rule_version table data column to mediumtext in mysql" 2024-09-17T00:17:29.470 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="add column for to alert_rule_version" 2024-09-17T00:17:29.470 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="add column annotations to alert_rule_version" 2024-09-17T00:17:29.470 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="add column labels to alert_rule_version" 2024-09-17T00:17:29.470 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id=create_alert_configuration_table 2024-09-17T00:17:29.471 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="Add column default in alert_configuration" 2024-09-17T00:17:29.471 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" 2024-09-17T00:17:29.471 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="add column org_id in alert_configuration" 2024-09-17T00:17:29.471 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_configuration table on org_id column" 2024-09-17T00:17:29.471 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id=create_ngalert_configuration_table 2024-09-17T00:17:29.471 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="add index in ngalert_configuration on org_id column" 2024-09-17T00:17:29.471 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="clear migration entry \"remove unified alerting data\"" 2024-09-17T00:17:29.471 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="move dashboard alerts to unified alerting" 2024-09-17T00:17:29.471 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="create library_element table v1" 2024-09-17T00:17:29.471 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="add index library_element org_id-folder_id-name-kind" 2024-09-17T00:17:29.471 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="create library_element_connection table v1" 2024-09-17T00:17:29.471 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="add index library_element_connection element_id-kind-connection_id" 2024-09-17T00:17:29.471 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index library_element org_id_uid" 2024-09-17T00:17:29.471 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="clone move dashboard alerts to unified alerting" 2024-09-17T00:17:29.472 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="create data_keys table" 2024-09-17T00:17:29.472 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="create kv_store table v1" 2024-09-17T00:17:29.472 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="add index kv_store.org_id-namespace-key" 2024-09-17T00:17:29.472 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="update dashboard_uid and panel_id from existing annotations" 2024-09-17T00:17:29.472 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="create permission table" 2024-09-17T00:17:29.472 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index permission.role_id" 2024-09-17T00:17:29.472 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index role_id_action_scope" 2024-09-17T00:17:29.472 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="create role table" 2024-09-17T00:17:29.472 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="add column display_name" 2024-09-17T00:17:29.472 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="add column group_name" 2024-09-17T00:17:29.472 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="add index role.org_id" 2024-09-17T00:17:29.472 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index role_org_id_name" 2024-09-17T00:17:29.472 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="add index role_org_id_uid" 2024-09-17T00:17:29.473 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="create team role table" 2024-09-17T00:17:29.473 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_role.org_id" 2024-09-17T00:17:29.473 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_role_org_id_team_id_role_id" 2024-09-17T00:17:29.473 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_role.team_id" 2024-09-17T00:17:29.473 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="create user role table" 2024-09-17T00:17:29.473 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="add index user_role.org_id" 2024-09-17T00:17:29.473 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_role_org_id_user_id_role_id" 2024-09-17T00:17:29.473 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="add index user_role.user_id" 2024-09-17T00:17:29.473 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="create builtin role table" 2024-09-17T00:17:29.473 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="add index builtin_role.role_id" 2024-09-17T00:17:29.473 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="add index builtin_role.name" 2024-09-17T00:17:29.473 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="Add column org_id to builtin_role table" 2024-09-17T00:17:29.473 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="add index builtin_role.org_id" 2024-09-17T00:17:29.473 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index builtin_role_org_id_role_id_role" 2024-09-17T00:17:29.474 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="Remove unique index role_org_id_uid" 2024-09-17T00:17:29.474 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index role.uid" 2024-09-17T00:17:29.474 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="create seed assignment table" 2024-09-17T00:17:29.474 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index builtin_role_role_name" 2024-09-17T00:17:29.474 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="migrations completed" logger=migrator performed=381 skipped=0 duration=565.406828ms 2024-09-17T00:17:29.474 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Created default organization" logger=sqlstore 2024-09-17T00:17:29.474 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Initialising plugins" logger=plugin.manager 2024-09-17T00:17:29.474 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=input 2024-09-17T00:17:29.474 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=grafana-piechart-panel 2024-09-17T00:17:29.474 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=vonage-status-panel 2024-09-17T00:17:29.717 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="Live Push Gateway initialization" logger=live.push_http 2024-09-17T00:17:29.717 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=warn msg="[Deprecated] the datasource provisioning config is outdated. please upgrade" logger=provisioning.datasources filename=/etc/grafana/provisioning/datasources/ceph-dashboard.yml 2024-09-17T00:17:29.717 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="inserting datasource from configuration " logger=provisioning.datasources name=Dashboard1 uid=P43CA22E17D0F9596 2024-09-17T00:17:29.717 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="warming cache for startup" logger=ngalert 2024-09-17T00:17:29.717 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="starting MultiOrg Alertmanager" logger=ngalert.multiorg.alertmanager 2024-09-17T00:17:29.717 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:17:29+0000 lvl=info msg="HTTP Server Listen" logger=http.server address=[::]:3000 protocol=https subUrl= socket= 2024-09-17T00:17:30.186 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:17:29 smithi002 ceph-mon[28203]: pgmap v42: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:17:30.187 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:17:29 smithi002 ceph-mon[23737]: pgmap v42: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:17:30.266 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:17:29 smithi099 ceph-mon[24594]: pgmap v42: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:17:31.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:17:31 smithi002 ceph-mon[23737]: pgmap v43: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:17:31.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:17:31 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:17:31.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:17:31 smithi002 ceph-mon[23737]: osdmap e61: 8 total, 8 up, 8 in 2024-09-17T00:17:31.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:17:31 smithi002 ceph-mon[23737]: from='client.? 172.21.15.2:0/1619313745' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-09-17T00:17:31.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:17:31 smithi002 ceph-mon[28203]: pgmap v43: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:17:31.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:17:31 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:17:31.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:17:31 smithi002 ceph-mon[28203]: osdmap e61: 8 total, 8 up, 8 in 2024-09-17T00:17:31.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:17:31 smithi002 ceph-mon[28203]: from='client.? 172.21.15.2:0/1619313745' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-09-17T00:17:32.019 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-09-17T00:17:32.019 INFO:teuthology.orchestra.run.smithi002.stdout: "id": "014fa0b8-4536-4b8c-891b-d533f09e574f", 2024-09-17T00:17:32.019 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "r", 2024-09-17T00:17:32.019 INFO:teuthology.orchestra.run.smithi002.stdout: "current_period": "a600141c-4d91-42fd-bee1-a35c2eecebf3", 2024-09-17T00:17:32.019 INFO:teuthology.orchestra.run.smithi002.stdout: "epoch": 1 2024-09-17T00:17:32.020 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-09-17T00:17:32.186 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:17:31 smithi099 ceph-mon[24594]: pgmap v43: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:17:32.186 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:17:31 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:17:32.187 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:17:31 smithi099 ceph-mon[24594]: osdmap e61: 8 total, 8 up, 8 in 2024-09-17T00:17:32.187 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:17:31 smithi099 ceph-mon[24594]: from='client.? 172.21.15.2:0/1619313745' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-09-17T00:17:32.723 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3aad27a2-7489-11ef-bceb-c7b262605968 -e sha1=c2ddf9721ef254645201cdbabd9c8f49c30a586f -- bash -c 'radosgw-admin zonegroup create --rgw-zonegroup=default --master --default' 2024-09-17T00:17:33.409 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:17:32 smithi002 ceph-mon[28203]: from='client.? 172.21.15.2:0/1619313745' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-09-17T00:17:33.409 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:17:32 smithi002 ceph-mon[28203]: osdmap e62: 8 total, 8 up, 8 in 2024-09-17T00:17:33.409 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:17:32 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:17:33.410 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:17:32 smithi002 ceph-mon[23737]: from='client.? 172.21.15.2:0/1619313745' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-09-17T00:17:33.410 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:17:32 smithi002 ceph-mon[23737]: osdmap e62: 8 total, 8 up, 8 in 2024-09-17T00:17:33.410 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:17:32 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:17:33.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:17:32 smithi099 ceph-mon[24594]: from='client.? 172.21.15.2:0/1619313745' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-09-17T00:17:33.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:17:32 smithi099 ceph-mon[24594]: osdmap e62: 8 total, 8 up, 8 in 2024-09-17T00:17:33.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:17:32 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:17:34.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:17:33 smithi099 ceph-mon[24594]: pgmap v46: 33 pgs: 32 creating+peering, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:17:34.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:17:33 smithi099 ceph-mon[24594]: osdmap e63: 8 total, 8 up, 8 in 2024-09-17T00:17:34.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:17:33 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:17:34.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:17:33 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:17:34.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:17:33 smithi002 ceph-mon[28203]: pgmap v46: 33 pgs: 32 creating+peering, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:17:34.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:17:33 smithi002 ceph-mon[28203]: osdmap e63: 8 total, 8 up, 8 in 2024-09-17T00:17:34.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:17:33 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:17:34.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:17:33 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:17:34.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:17:33 smithi002 ceph-mon[23737]: pgmap v46: 33 pgs: 32 creating+peering, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:17:34.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:17:33 smithi002 ceph-mon[23737]: osdmap e63: 8 total, 8 up, 8 in 2024-09-17T00:17:34.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:17:33 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:17:34.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:17:33 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:17:35.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:17:35 smithi099 ceph-mon[24594]: Reconfiguring alertmanager.a (dependencies changed)... 2024-09-17T00:17:35.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:17:35 smithi099 ceph-mon[24594]: Reconfiguring daemon alertmanager.a on smithi002 2024-09-17T00:17:35.217 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:17:34 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[25772]: ::ffff:172.21.15.99 - - [17/Sep/2024:00:17:34] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-09-17T00:17:35.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:17:35 smithi002 ceph-mon[28203]: Reconfiguring alertmanager.a (dependencies changed)... 2024-09-17T00:17:35.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:17:35 smithi002 ceph-mon[28203]: Reconfiguring daemon alertmanager.a on smithi002 2024-09-17T00:17:35.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:17:35 smithi002 ceph-mon[23737]: Reconfiguring alertmanager.a (dependencies changed)... 2024-09-17T00:17:35.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:17:35 smithi002 ceph-mon[23737]: Reconfiguring daemon alertmanager.a on smithi002 2024-09-17T00:17:35.909 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-09-17T00:17:35.910 INFO:teuthology.orchestra.run.smithi002.stdout: "id": "91ff7d2e-24a7-4ed1-aedf-e28741f6a264", 2024-09-17T00:17:35.910 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "default", 2024-09-17T00:17:35.910 INFO:teuthology.orchestra.run.smithi002.stdout: "api_name": "default", 2024-09-17T00:17:35.910 INFO:teuthology.orchestra.run.smithi002.stdout: "is_master": "true", 2024-09-17T00:17:35.910 INFO:teuthology.orchestra.run.smithi002.stdout: "endpoints": [], 2024-09-17T00:17:35.910 INFO:teuthology.orchestra.run.smithi002.stdout: "hostnames": [], 2024-09-17T00:17:35.910 INFO:teuthology.orchestra.run.smithi002.stdout: "hostnames_s3website": [], 2024-09-17T00:17:35.910 INFO:teuthology.orchestra.run.smithi002.stdout: "master_zone": "", 2024-09-17T00:17:35.910 INFO:teuthology.orchestra.run.smithi002.stdout: "zones": [], 2024-09-17T00:17:35.910 INFO:teuthology.orchestra.run.smithi002.stdout: "placement_targets": [], 2024-09-17T00:17:35.910 INFO:teuthology.orchestra.run.smithi002.stdout: "default_placement": "", 2024-09-17T00:17:35.910 INFO:teuthology.orchestra.run.smithi002.stdout: "realm_id": "014fa0b8-4536-4b8c-891b-d533f09e574f", 2024-09-17T00:17:35.910 INFO:teuthology.orchestra.run.smithi002.stdout: "sync_policy": { 2024-09-17T00:17:35.910 INFO:teuthology.orchestra.run.smithi002.stdout: "groups": [] 2024-09-17T00:17:35.911 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-09-17T00:17:35.911 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-09-17T00:17:36.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:17:36 smithi002 ceph-mon[28203]: pgmap v48: 33 pgs: 32 creating+peering, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:17:36.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:17:36 smithi002 ceph-mon[23737]: pgmap v48: 33 pgs: 32 creating+peering, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:17:36.466 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:17:36 smithi099 ceph-mon[24594]: pgmap v48: 33 pgs: 32 creating+peering, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:17:36.564 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3aad27a2-7489-11ef-bceb-c7b262605968 -e sha1=c2ddf9721ef254645201cdbabd9c8f49c30a586f -- bash -c 'radosgw-admin zone create --rgw-zonegroup=default --rgw-zone=z --master --default' 2024-09-17T00:17:38.323 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:17:38 smithi002 ceph-mon[23737]: pgmap v49: 33 pgs: 26 creating+peering, 7 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 170 B/s wr, 0 op/s 2024-09-17T00:17:38.323 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:17:38 smithi002 ceph-mon[28203]: pgmap v49: 33 pgs: 26 creating+peering, 7 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 170 B/s wr, 0 op/s 2024-09-17T00:17:38.466 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:17:38 smithi099 ceph-mon[24594]: pgmap v49: 33 pgs: 26 creating+peering, 7 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 170 B/s wr, 0 op/s 2024-09-17T00:17:38.636 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Sep 17 00:17:38 smithi002 systemd[1]: Stopping Ceph alertmanager.a for 3aad27a2-7489-11ef-bceb-c7b262605968... 2024-09-17T00:17:38.894 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Sep 17 00:17:38 smithi002 bash[47775]: Error: no container with name or ID "ceph-3aad27a2-7489-11ef-bceb-c7b262605968-alertmanager.a" found: no such container 2024-09-17T00:17:38.894 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Sep 17 00:17:38 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-alertmanager-a[46262]: level=info ts=2024-09-17T00:17:38.773Z caller=main.go:557 msg="Received SIGTERM, exiting gracefully..." 2024-09-17T00:17:39.229 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Sep 17 00:17:38 smithi002 podman[47788]: 2024-09-17 00:17:38.898170747 +0000 UTC m=+0.246636194 container died 7c35d04fcfe4066f9a6d8928d1a58dd6239e09ef450878894772070b9f00283b (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-17T00:17:39.229 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:17:39 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: ::ffff:172.21.15.99 - - [17/Sep/2024:00:17:39] "GET /metrics HTTP/1.1" 200 191469 "" "Prometheus/2.33.4" 2024-09-17T00:17:39.661 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Sep 17 00:17:39 smithi002 podman[47788]: 2024-09-17 00:17:39.40798931 +0000 UTC m=+0.756454786 container cleanup 7c35d04fcfe4066f9a6d8928d1a58dd6239e09ef450878894772070b9f00283b (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-17T00:17:39.661 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Sep 17 00:17:39 smithi002 bash[47788]: ceph-3aad27a2-7489-11ef-bceb-c7b262605968-alertmanager-a 2024-09-17T00:17:40.262 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:17:40 smithi002 ceph-mon[28203]: pgmap v50: 33 pgs: 33 active+clean; 578 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 555 B/s rd, 1.5 KiB/s wr, 2 op/s 2024-09-17T00:17:40.263 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Sep 17 00:17:40 smithi002 podman[47802]: 2024-09-17 00:17:40.060240596 +0000 UTC m=+1.264518915 container remove 7c35d04fcfe4066f9a6d8928d1a58dd6239e09ef450878894772070b9f00283b (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-17T00:17:40.263 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Sep 17 00:17:40 smithi002 podman[47802]: 2024-09-17 00:17:40.202707841 +0000 UTC m=+1.406986160 volume remove f4329900baca96a599cf7e73b3234db1b86b382c8d6d8de42f9cf1bed00cc46d 2024-09-17T00:17:40.263 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:17:40 smithi002 ceph-mon[23737]: pgmap v50: 33 pgs: 33 active+clean; 578 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 555 B/s rd, 1.5 KiB/s wr, 2 op/s 2024-09-17T00:17:40.466 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:17:40 smithi099 ceph-mon[24594]: pgmap v50: 33 pgs: 33 active+clean; 578 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 555 B/s rd, 1.5 KiB/s wr, 2 op/s 2024-09-17T00:17:40.627 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Sep 17 00:17:40 smithi002 bash[47829]: Error: no container with name or ID "ceph-3aad27a2-7489-11ef-bceb-c7b262605968-alertmanager.a" found: no such container 2024-09-17T00:17:40.627 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Sep 17 00:17:40 smithi002 systemd[1]: ceph-3aad27a2-7489-11ef-bceb-c7b262605968@alertmanager.a.service: Deactivated successfully. 2024-09-17T00:17:40.627 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Sep 17 00:17:40 smithi002 systemd[1]: Stopped Ceph alertmanager.a for 3aad27a2-7489-11ef-bceb-c7b262605968. 2024-09-17T00:17:40.627 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Sep 17 00:17:40 smithi002 systemd[1]: Starting Ceph alertmanager.a for 3aad27a2-7489-11ef-bceb-c7b262605968... 2024-09-17T00:17:41.163 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-09-17T00:17:41.163 INFO:teuthology.orchestra.run.smithi002.stdout: "id": "cec51b2d-fa24-475f-91d4-d3238f0fb553", 2024-09-17T00:17:41.164 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "z", 2024-09-17T00:17:41.164 INFO:teuthology.orchestra.run.smithi002.stdout: "domain_root": "z.rgw.meta:root", 2024-09-17T00:17:41.164 INFO:teuthology.orchestra.run.smithi002.stdout: "control_pool": "z.rgw.control", 2024-09-17T00:17:41.164 INFO:teuthology.orchestra.run.smithi002.stdout: "gc_pool": "z.rgw.log:gc", 2024-09-17T00:17:41.164 INFO:teuthology.orchestra.run.smithi002.stdout: "lc_pool": "z.rgw.log:lc", 2024-09-17T00:17:41.164 INFO:teuthology.orchestra.run.smithi002.stdout: "log_pool": "z.rgw.log", 2024-09-17T00:17:41.164 INFO:teuthology.orchestra.run.smithi002.stdout: "intent_log_pool": "z.rgw.log:intent", 2024-09-17T00:17:41.165 INFO:teuthology.orchestra.run.smithi002.stdout: "usage_log_pool": "z.rgw.log:usage", 2024-09-17T00:17:41.165 INFO:teuthology.orchestra.run.smithi002.stdout: "roles_pool": "z.rgw.meta:roles", 2024-09-17T00:17:41.165 INFO:teuthology.orchestra.run.smithi002.stdout: "reshard_pool": "z.rgw.log:reshard", 2024-09-17T00:17:41.165 INFO:teuthology.orchestra.run.smithi002.stdout: "user_keys_pool": "z.rgw.meta:users.keys", 2024-09-17T00:17:41.165 INFO:teuthology.orchestra.run.smithi002.stdout: "user_email_pool": "z.rgw.meta:users.email", 2024-09-17T00:17:41.165 INFO:teuthology.orchestra.run.smithi002.stdout: "user_swift_pool": "z.rgw.meta:users.swift", 2024-09-17T00:17:41.165 INFO:teuthology.orchestra.run.smithi002.stdout: "user_uid_pool": "z.rgw.meta:users.uid", 2024-09-17T00:17:41.165 INFO:teuthology.orchestra.run.smithi002.stdout: "otp_pool": "z.rgw.otp", 2024-09-17T00:17:41.165 INFO:teuthology.orchestra.run.smithi002.stdout: "system_key": { 2024-09-17T00:17:41.166 INFO:teuthology.orchestra.run.smithi002.stdout: "access_key": "", 2024-09-17T00:17:41.166 INFO:teuthology.orchestra.run.smithi002.stdout: "secret_key": "" 2024-09-17T00:17:41.166 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-09-17T00:17:41.166 INFO:teuthology.orchestra.run.smithi002.stdout: "placement_pools": [ 2024-09-17T00:17:41.166 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-09-17T00:17:41.166 INFO:teuthology.orchestra.run.smithi002.stdout: "key": "default-placement", 2024-09-17T00:17:41.166 INFO:teuthology.orchestra.run.smithi002.stdout: "val": { 2024-09-17T00:17:41.166 INFO:teuthology.orchestra.run.smithi002.stdout: "index_pool": "z.rgw.buckets.index", 2024-09-17T00:17:41.167 INFO:teuthology.orchestra.run.smithi002.stdout: "storage_classes": { 2024-09-17T00:17:41.167 INFO:teuthology.orchestra.run.smithi002.stdout: "STANDARD": { 2024-09-17T00:17:41.167 INFO:teuthology.orchestra.run.smithi002.stdout: "data_pool": "z.rgw.buckets.data" 2024-09-17T00:17:41.167 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-09-17T00:17:41.167 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-09-17T00:17:41.167 INFO:teuthology.orchestra.run.smithi002.stdout: "data_extra_pool": "z.rgw.buckets.non-ec", 2024-09-17T00:17:41.167 INFO:teuthology.orchestra.run.smithi002.stdout: "index_type": 0 2024-09-17T00:17:41.167 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-09-17T00:17:41.168 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-09-17T00:17:41.168 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-09-17T00:17:41.168 INFO:teuthology.orchestra.run.smithi002.stdout: "realm_id": "014fa0b8-4536-4b8c-891b-d533f09e574f", 2024-09-17T00:17:41.168 INFO:teuthology.orchestra.run.smithi002.stdout: "notif_pool": "z.rgw.log:notif" 2024-09-17T00:17:41.168 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-09-17T00:17:41.919 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3aad27a2-7489-11ef-bceb-c7b262605968 -e sha1=c2ddf9721ef254645201cdbabd9c8f49c30a586f -- bash -c 'radosgw-admin period update --rgw-realm=r --commit' 2024-09-17T00:17:42.126 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:17:42 smithi002 ceph-mon[28203]: pgmap v51: 33 pgs: 33 active+clean; 578 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 489 B/s rd, 1.3 KiB/s wr, 2 op/s 2024-09-17T00:17:42.127 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:17:42 smithi002 ceph-mon[23737]: pgmap v51: 33 pgs: 33 active+clean; 578 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 489 B/s rd, 1.3 KiB/s wr, 2 op/s 2024-09-17T00:17:42.127 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Sep 17 00:17:42 smithi002 podman[47999]: 2024-09-17 00:17:41.923023358 +0000 UTC m=+0.020842536 image pull ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4 quay.io/prometheus/alertmanager:v0.23.0 2024-09-17T00:17:42.127 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Sep 17 00:17:42 smithi002 podman[47999]: 2024-09-17 00:17:42.03808014 +0000 UTC m=+0.135899308 volume create e47b4d4b018bf8c90622b1043393783f76c44a782c3d54c4f43f9f264f2172cf 2024-09-17T00:17:42.127 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Sep 17 00:17:42 smithi002 podman[47999]: 2024-09-17 00:17:42.130246318 +0000 UTC m=+0.228065550 container create f8bd5774d3248d7796a45c11b454ab89f567b72b0bc0b7ed04f7eeb2e24f06ed (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-17T00:17:42.466 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:17:42 smithi099 ceph-mon[24594]: pgmap v51: 33 pgs: 33 active+clean; 578 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 489 B/s rd, 1.3 KiB/s wr, 2 op/s 2024-09-17T00:17:42.729 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Sep 17 00:17:42 smithi002 podman[47999]: 2024-09-17 00:17:42.449677395 +0000 UTC m=+0.547496562 container init f8bd5774d3248d7796a45c11b454ab89f567b72b0bc0b7ed04f7eeb2e24f06ed (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-17T00:17:42.729 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Sep 17 00:17:42 smithi002 podman[47999]: 2024-09-17 00:17:42.452736777 +0000 UTC m=+0.550555954 container start f8bd5774d3248d7796a45c11b454ab89f567b72b0bc0b7ed04f7eeb2e24f06ed (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-17T00:17:42.730 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Sep 17 00:17:42 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-alertmanager-a[48044]: level=info ts=2024-09-17T00:17:42.463Z caller=main.go:225 msg="Starting Alertmanager" version="(version=0.23.0, branch=HEAD, revision=61046b17771a57cfd4c4a51be370ab930a4d7d54)" 2024-09-17T00:17:42.730 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Sep 17 00:17:42 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-alertmanager-a[48044]: level=info ts=2024-09-17T00:17:42.463Z caller=main.go:226 build_context="(go=go1.16.7, user=root@e21a959be8d2, date=20210825-10:48:55)" 2024-09-17T00:17:42.730 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Sep 17 00:17:42 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-alertmanager-a[48044]: level=info ts=2024-09-17T00:17:42.464Z caller=cluster.go:184 component=cluster msg="setting advertise address explicitly" addr=172.21.15.2 port=9094 2024-09-17T00:17:42.730 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Sep 17 00:17:42 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-alertmanager-a[48044]: level=info ts=2024-09-17T00:17:42.465Z caller=cluster.go:671 component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-09-17T00:17:42.730 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Sep 17 00:17:42 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-alertmanager-a[48044]: level=info ts=2024-09-17T00:17:42.491Z caller=coordinator.go:113 component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-17T00:17:42.730 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Sep 17 00:17:42 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-alertmanager-a[48044]: level=info ts=2024-09-17T00:17:42.491Z caller=coordinator.go:126 component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-17T00:17:42.730 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Sep 17 00:17:42 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-alertmanager-a[48044]: level=info ts=2024-09-17T00:17:42.493Z caller=main.go:518 msg=Listening address=:9093 2024-09-17T00:17:42.730 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Sep 17 00:17:42 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-alertmanager-a[48044]: level=info ts=2024-09-17T00:17:42.493Z caller=tls_config.go:191 msg="TLS is disabled." http2=false 2024-09-17T00:17:42.730 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Sep 17 00:17:42 smithi002 bash[47999]: f8bd5774d3248d7796a45c11b454ab89f567b72b0bc0b7ed04f7eeb2e24f06ed 2024-09-17T00:17:42.730 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Sep 17 00:17:42 smithi002 systemd[1]: Started Ceph alertmanager.a for 3aad27a2-7489-11ef-bceb-c7b262605968. 2024-09-17T00:17:43.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:17:43 smithi002 ceph-mon[28203]: pgmap v52: 33 pgs: 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 716 B/s rd, 1.6 KiB/s wr, 2 op/s 2024-09-17T00:17:43.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:17:43 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:17:43.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:17:43 smithi002 ceph-mon[28203]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-17T00:17:43.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:17:43 smithi002 ceph-mon[28203]: Reconfiguring daemon prometheus.a on smithi099 2024-09-17T00:17:43.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:17:43 smithi002 ceph-mon[23737]: pgmap v52: 33 pgs: 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 716 B/s rd, 1.6 KiB/s wr, 2 op/s 2024-09-17T00:17:43.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:17:43 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:17:43.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:17:43 smithi002 ceph-mon[23737]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-17T00:17:43.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:17:43 smithi002 ceph-mon[23737]: Reconfiguring daemon prometheus.a on smithi099 2024-09-17T00:17:43.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:17:43 smithi099 ceph-mon[24594]: pgmap v52: 33 pgs: 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 716 B/s rd, 1.6 KiB/s wr, 2 op/s 2024-09-17T00:17:43.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:17:43 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:17:43.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:17:43 smithi099 ceph-mon[24594]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-17T00:17:43.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:17:43 smithi099 ceph-mon[24594]: Reconfiguring daemon prometheus.a on smithi099 2024-09-17T00:17:44.729 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Sep 17 00:17:44 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-alertmanager-a[48044]: level=info ts=2024-09-17T00:17:44.466Z caller=cluster.go:696 component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000438s 2024-09-17T00:17:45.217 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:17:44 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[25772]: ::ffff:172.21.15.99 - - [17/Sep/2024:00:17:44] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-09-17T00:17:45.658 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:17:45 smithi099 systemd[1]: Stopping Ceph prometheus.a for 3aad27a2-7489-11ef-bceb-c7b262605968... 2024-09-17T00:17:45.659 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:17:45 smithi099 bash[41346]: Error: no container with name or ID "ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus.a" found: no such container 2024-09-17T00:17:45.959 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:17:45 smithi099 ceph-mon[24594]: pgmap v53: 33 pgs: 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 630 B/s rd, 1.4 KiB/s wr, 2 op/s 2024-09-17T00:17:45.959 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:17:45 smithi099 ceph-mon[24594]: osdmap e64: 8 total, 8 up, 8 in 2024-09-17T00:17:45.959 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:17:45 smithi099 ceph-mon[24594]: from='client.? 172.21.15.2:0/1980305902' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-09-17T00:17:45.960 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:17:45 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[39646]: ts=2024-09-17T00:17:45.662Z caller=main.go:775 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-09-17T00:17:45.960 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:17:45 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[39646]: ts=2024-09-17T00:17:45.662Z caller=main.go:798 level=info msg="Stopping scrape discovery manager..." 2024-09-17T00:17:45.960 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:17:45 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[39646]: ts=2024-09-17T00:17:45.662Z caller=main.go:812 level=info msg="Stopping notify discovery manager..." 2024-09-17T00:17:45.960 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:17:45 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[39646]: ts=2024-09-17T00:17:45.662Z caller=main.go:834 level=info msg="Stopping scrape manager..." 2024-09-17T00:17:45.960 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:17:45 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[39646]: ts=2024-09-17T00:17:45.662Z caller=main.go:794 level=info msg="Scrape discovery manager stopped" 2024-09-17T00:17:45.960 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:17:45 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[39646]: ts=2024-09-17T00:17:45.662Z caller=main.go:808 level=info msg="Notify discovery manager stopped" 2024-09-17T00:17:45.960 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:17:45 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[39646]: ts=2024-09-17T00:17:45.662Z caller=manager.go:945 level=info component="rule manager" msg="Stopping rule manager..." 2024-09-17T00:17:45.960 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:17:45 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[39646]: ts=2024-09-17T00:17:45.662Z caller=manager.go:955 level=info component="rule manager" msg="Rule manager stopped" 2024-09-17T00:17:45.961 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:17:45 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[39646]: ts=2024-09-17T00:17:45.662Z caller=main.go:828 level=info msg="Scrape manager stopped" 2024-09-17T00:17:45.961 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:17:45 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[39646]: ts=2024-09-17T00:17:45.662Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..." 2024-09-17T00:17:45.961 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:17:45 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[39646]: ts=2024-09-17T00:17:45.663Z caller=main.go:1054 level=info msg="Notifier manager stopped" 2024-09-17T00:17:45.961 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:17:45 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[39646]: ts=2024-09-17T00:17:45.663Z caller=main.go:1066 level=info msg="See you next time!" 2024-09-17T00:17:45.961 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:17:45 smithi099 podman[41357]: 2024-09-17 00:17:45.672443725 +0000 UTC m=+0.110124648 container died e644918fd5d891ae7fd53859ff4a91611041c601a3076facc6422c54d193a98b (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-17T00:17:45.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:17:45 smithi002 ceph-mon[28203]: pgmap v53: 33 pgs: 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 630 B/s rd, 1.4 KiB/s wr, 2 op/s 2024-09-17T00:17:45.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:17:45 smithi002 ceph-mon[28203]: osdmap e64: 8 total, 8 up, 8 in 2024-09-17T00:17:45.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:17:45 smithi002 ceph-mon[28203]: from='client.? 172.21.15.2:0/1980305902' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-09-17T00:17:45.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:17:45 smithi002 ceph-mon[23737]: pgmap v53: 33 pgs: 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 630 B/s rd, 1.4 KiB/s wr, 2 op/s 2024-09-17T00:17:45.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:17:45 smithi002 ceph-mon[23737]: osdmap e64: 8 total, 8 up, 8 in 2024-09-17T00:17:45.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:17:45 smithi002 ceph-mon[23737]: from='client.? 172.21.15.2:0/1980305902' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-09-17T00:17:46.216 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:17:46 smithi099 podman[41357]: 2024-09-17 00:17:46.053795686 +0000 UTC m=+0.491476601 container cleanup e644918fd5d891ae7fd53859ff4a91611041c601a3076facc6422c54d193a98b (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-17T00:17:46.216 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:17:46 smithi099 bash[41357]: ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a 2024-09-17T00:17:46.217 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:17:46 smithi099 bash[41384]: Error: no container with name or ID "ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus.a" found: no such container 2024-09-17T00:17:46.544 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:17:46 smithi099 podman[41371]: 2024-09-17 00:17:46.296791205 +0000 UTC m=+0.625856559 container remove e644918fd5d891ae7fd53859ff4a91611041c601a3076facc6422c54d193a98b (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-17T00:17:46.544 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:17:46 smithi099 systemd[1]: ceph-3aad27a2-7489-11ef-bceb-c7b262605968@prometheus.a.service: Deactivated successfully. 2024-09-17T00:17:46.544 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:17:46 smithi099 systemd[1]: Stopped Ceph prometheus.a for 3aad27a2-7489-11ef-bceb-c7b262605968. 2024-09-17T00:17:46.544 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:17:46 smithi099 systemd[1]: Starting Ceph prometheus.a for 3aad27a2-7489-11ef-bceb-c7b262605968... 2024-09-17T00:17:46.544 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:17:46 smithi099 podman[41450]: 2024-09-17 00:17:46.427901184 +0000 UTC m=+0.017335075 image pull 514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d quay.io/prometheus/prometheus:v2.33.4 2024-09-17T00:17:46.544 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:17:46 smithi099 podman[41450]: 2024-09-17 00:17:46.547691845 +0000 UTC m=+0.137125722 container create a1529c2c7a1cafe7d2add603f09411f6a07e99ebd29fd45d99dc5adbc76ca919 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-17T00:17:46.794 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:17:46 smithi099 ceph-mon[24594]: from='client.? 172.21.15.2:0/1980305902' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]': finished 2024-09-17T00:17:46.794 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:17:46 smithi099 ceph-mon[24594]: osdmap e65: 8 total, 8 up, 8 in 2024-09-17T00:17:46.795 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:17:46 smithi099 podman[41450]: 2024-09-17 00:17:46.766968092 +0000 UTC m=+0.356401975 container init a1529c2c7a1cafe7d2add603f09411f6a07e99ebd29fd45d99dc5adbc76ca919 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-17T00:17:46.795 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:17:46 smithi099 podman[41450]: 2024-09-17 00:17:46.769865671 +0000 UTC m=+0.359299552 container start a1529c2c7a1cafe7d2add603f09411f6a07e99ebd29fd45d99dc5adbc76ca919 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-17T00:17:46.795 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:17:46 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[41464]: ts=2024-09-17T00:17:46.793Z caller=main.go:475 level=info msg="No time or size retention was set so using the default time retention" duration=15d 2024-09-17T00:17:46.795 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:17:46 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[41464]: ts=2024-09-17T00:17:46.793Z caller=main.go:512 level=info msg="Starting Prometheus" version="(version=2.33.4, branch=HEAD, revision=83032011a5d3e6102624fe58241a374a7201fee8)" 2024-09-17T00:17:46.795 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:17:46 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[41464]: ts=2024-09-17T00:17:46.793Z caller=main.go:517 level=info build_context="(go=go1.17.7, user=root@d13bf69e7be8, date=20220222-16:51:28)" 2024-09-17T00:17:46.795 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:17:46 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[41464]: ts=2024-09-17T00:17:46.793Z caller=main.go:518 level=info host_details="(Linux 5.14.0-508.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024 x86_64 smithi099 (none))" 2024-09-17T00:17:46.795 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:17:46 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[41464]: ts=2024-09-17T00:17:46.793Z caller=main.go:519 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-09-17T00:17:46.795 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:17:46 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[41464]: ts=2024-09-17T00:17:46.793Z caller=main.go:520 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-09-17T00:17:46.795 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:17:46 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[41464]: ts=2024-09-17T00:17:46.794Z caller=web.go:570 level=info component=web msg="Start listening for connections" address=:9095 2024-09-17T00:17:46.795 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:17:46 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[41464]: ts=2024-09-17T00:17:46.795Z caller=main.go:923 level=info msg="Starting TSDB ..." 2024-09-17T00:17:46.795 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:17:46 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[41464]: ts=2024-09-17T00:17:46.796Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false 2024-09-17T00:17:46.796 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:17:46 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[41464]: ts=2024-09-17T00:17:46.798Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-09-17T00:17:46.796 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:17:46 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[41464]: ts=2024-09-17T00:17:46.798Z caller=head.go:527 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=2.449µs 2024-09-17T00:17:46.796 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:17:46 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[41464]: ts=2024-09-17T00:17:46.798Z caller=head.go:533 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-09-17T00:17:46.983 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:17:46 smithi002 ceph-mon[28203]: from='client.? 172.21.15.2:0/1980305902' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]': finished 2024-09-17T00:17:46.983 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:17:46 smithi002 ceph-mon[28203]: osdmap e65: 8 total, 8 up, 8 in 2024-09-17T00:17:46.983 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:17:46 smithi002 ceph-mon[23737]: from='client.? 172.21.15.2:0/1980305902' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]': finished 2024-09-17T00:17:46.983 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:17:46 smithi002 ceph-mon[23737]: osdmap e65: 8 total, 8 up, 8 in 2024-09-17T00:17:47.217 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:17:46 smithi099 bash[41450]: a1529c2c7a1cafe7d2add603f09411f6a07e99ebd29fd45d99dc5adbc76ca919 2024-09-17T00:17:47.217 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:17:46 smithi099 systemd[1]: Started Ceph prometheus.a for 3aad27a2-7489-11ef-bceb-c7b262605968. 2024-09-17T00:17:47.217 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:17:46 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[41464]: ts=2024-09-17T00:17:46.989Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=1 2024-09-17T00:17:47.217 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:17:46 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[41464]: ts=2024-09-17T00:17:46.990Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=1 2024-09-17T00:17:47.217 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:17:46 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[41464]: ts=2024-09-17T00:17:46.990Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=33.532µs wal_replay_duration=192.276291ms total_replay_duration=192.330537ms 2024-09-17T00:17:47.217 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:17:46 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[41464]: ts=2024-09-17T00:17:46.992Z caller=main.go:944 level=info fs_type=XFS_SUPER_MAGIC 2024-09-17T00:17:47.217 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:17:46 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[41464]: ts=2024-09-17T00:17:46.992Z caller=main.go:947 level=info msg="TSDB started" 2024-09-17T00:17:47.217 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:17:46 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[41464]: ts=2024-09-17T00:17:46.992Z caller=main.go:1128 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-09-17T00:17:47.218 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:17:47 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[41464]: ts=2024-09-17T00:17:47.009Z caller=main.go:1165 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=17.608143ms db_storage=684ns remote_storage=1.302µs web_handler=367ns query_engine=720ns scrape=394.374µs scrape_sd=31.08µs notify=27.178µs notify_sd=7.098µs rules=16.865521ms 2024-09-17T00:17:47.218 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:17:47 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[41464]: ts=2024-09-17T00:17:47.009Z caller=main.go:896 level=info msg="Server is ready to receive web requests." 2024-09-17T00:17:48.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:17:47 smithi099 ceph-mon[24594]: pgmap v56: 65 pgs: 17 unknown, 48 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 639 B/s wr, 3 op/s 2024-09-17T00:17:48.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:17:47 smithi099 ceph-mon[24594]: osdmap e66: 8 total, 8 up, 8 in 2024-09-17T00:17:48.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:17:47 smithi099 ceph-mon[24594]: from='client.? 172.21.15.2:0/1980305902' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-09-17T00:17:48.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:17:47 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:17:48.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:17:47 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-17T00:17:48.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:17:47 smithi099 ceph-mon[24594]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-17T00:17:48.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:17:47 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.2:9093"}]: dispatch 2024-09-17T00:17:48.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:17:47 smithi099 ceph-mon[24594]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.2:9093"}]: dispatch 2024-09-17T00:17:48.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:17:47 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:17:48.218 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:17:47 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-17T00:17:48.218 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:17:47 smithi099 ceph-mon[24594]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-17T00:17:48.218 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:17:47 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.99:9095"}]: dispatch 2024-09-17T00:17:48.218 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:17:47 smithi099 ceph-mon[24594]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.99:9095"}]: dispatch 2024-09-17T00:17:48.218 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:17:47 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:17:48.218 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:17:47 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-17T00:17:48.218 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:17:47 smithi099 ceph-mon[24594]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-17T00:17:48.218 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:17:47 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.99:3000"}]: dispatch 2024-09-17T00:17:48.218 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:17:47 smithi099 ceph-mon[24594]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.99:3000"}]: dispatch 2024-09-17T00:17:48.218 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:17:47 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:17:48.218 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:17:47 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:17:48.218 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:17:47 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:17:48.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:17:47 smithi002 ceph-mon[28203]: pgmap v56: 65 pgs: 17 unknown, 48 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 639 B/s wr, 3 op/s 2024-09-17T00:17:48.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:17:47 smithi002 ceph-mon[28203]: osdmap e66: 8 total, 8 up, 8 in 2024-09-17T00:17:48.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:17:47 smithi002 ceph-mon[28203]: from='client.? 172.21.15.2:0/1980305902' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-09-17T00:17:48.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:17:47 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:17:48.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:17:47 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-17T00:17:48.230 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:17:47 smithi002 ceph-mon[28203]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-17T00:17:48.230 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:17:47 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.2:9093"}]: dispatch 2024-09-17T00:17:48.230 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:17:47 smithi002 ceph-mon[28203]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.2:9093"}]: dispatch 2024-09-17T00:17:48.230 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:17:47 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:17:48.230 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:17:47 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-17T00:17:48.230 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:17:47 smithi002 ceph-mon[28203]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-17T00:17:48.230 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:17:47 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.99:9095"}]: dispatch 2024-09-17T00:17:48.230 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:17:47 smithi002 ceph-mon[28203]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.99:9095"}]: dispatch 2024-09-17T00:17:48.230 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:17:47 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:17:48.230 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:17:47 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-17T00:17:48.230 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:17:47 smithi002 ceph-mon[28203]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-17T00:17:48.230 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:17:47 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.99:3000"}]: dispatch 2024-09-17T00:17:48.230 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:17:47 smithi002 ceph-mon[28203]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.99:3000"}]: dispatch 2024-09-17T00:17:48.230 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:17:47 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:17:48.231 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:17:47 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:17:48.231 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:17:47 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:17:48.231 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:17:47 smithi002 ceph-mon[23737]: pgmap v56: 65 pgs: 17 unknown, 48 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 639 B/s wr, 3 op/s 2024-09-17T00:17:48.231 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:17:47 smithi002 ceph-mon[23737]: osdmap e66: 8 total, 8 up, 8 in 2024-09-17T00:17:48.231 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:17:47 smithi002 ceph-mon[23737]: from='client.? 172.21.15.2:0/1980305902' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-09-17T00:17:48.231 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:17:47 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:17:48.231 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:17:47 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-17T00:17:48.231 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:17:47 smithi002 ceph-mon[23737]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-17T00:17:48.231 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:17:47 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.2:9093"}]: dispatch 2024-09-17T00:17:48.231 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:17:47 smithi002 ceph-mon[23737]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.2:9093"}]: dispatch 2024-09-17T00:17:48.231 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:17:47 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:17:48.232 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:17:47 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-17T00:17:48.232 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:17:47 smithi002 ceph-mon[23737]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-17T00:17:48.232 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:17:47 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.99:9095"}]: dispatch 2024-09-17T00:17:48.232 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:17:47 smithi002 ceph-mon[23737]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.99:9095"}]: dispatch 2024-09-17T00:17:48.232 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:17:47 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:17:48.232 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:17:47 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-17T00:17:48.232 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:17:47 smithi002 ceph-mon[23737]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-17T00:17:48.232 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:17:47 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.99:3000"}]: dispatch 2024-09-17T00:17:48.232 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:17:47 smithi002 ceph-mon[23737]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.99:3000"}]: dispatch 2024-09-17T00:17:48.232 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:17:47 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:17:48.232 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:17:47 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:17:48.232 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:17:47 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:17:49.017 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:17:48 smithi099 ceph-mon[24594]: from='client.? 172.21.15.2:0/1980305902' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]': finished 2024-09-17T00:17:49.018 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:17:48 smithi099 ceph-mon[24594]: osdmap e67: 8 total, 8 up, 8 in 2024-09-17T00:17:49.139 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:17:48 smithi002 ceph-mon[28203]: from='client.? 172.21.15.2:0/1980305902' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]': finished 2024-09-17T00:17:49.139 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:17:48 smithi002 ceph-mon[28203]: osdmap e67: 8 total, 8 up, 8 in 2024-09-17T00:17:49.139 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:17:48 smithi002 ceph-mon[23737]: from='client.? 172.21.15.2:0/1980305902' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]': finished 2024-09-17T00:17:49.139 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:17:48 smithi002 ceph-mon[23737]: osdmap e67: 8 total, 8 up, 8 in 2024-09-17T00:17:49.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:17:49 smithi099 ceph-mon[24594]: pgmap v59: 97 pgs: 32 unknown, 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 5.7 KiB/s rd, 5 op/s 2024-09-17T00:17:49.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:17:49 smithi099 ceph-mon[24594]: osdmap e68: 8 total, 8 up, 8 in 2024-09-17T00:17:49.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:17:49 smithi099 ceph-mon[24594]: from='client.? 172.21.15.2:0/3113681048' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-09-17T00:17:49.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:17:49 smithi099 ceph-mon[24594]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-09-17T00:17:49.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:17:49 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:17:50.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:17:49 smithi002 ceph-mon[23737]: pgmap v59: 97 pgs: 32 unknown, 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 5.7 KiB/s rd, 5 op/s 2024-09-17T00:17:50.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:17:49 smithi002 ceph-mon[23737]: osdmap e68: 8 total, 8 up, 8 in 2024-09-17T00:17:50.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:17:49 smithi002 ceph-mon[23737]: from='client.? 172.21.15.2:0/3113681048' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-09-17T00:17:50.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:17:49 smithi002 ceph-mon[23737]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-09-17T00:17:50.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:17:49 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:17:50.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:17:49 smithi002 ceph-mon[28203]: pgmap v59: 97 pgs: 32 unknown, 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 5.7 KiB/s rd, 5 op/s 2024-09-17T00:17:50.230 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:17:49 smithi002 ceph-mon[28203]: osdmap e68: 8 total, 8 up, 8 in 2024-09-17T00:17:50.230 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:17:49 smithi002 ceph-mon[28203]: from='client.? 172.21.15.2:0/3113681048' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-09-17T00:17:50.230 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:17:49 smithi002 ceph-mon[28203]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-09-17T00:17:50.230 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:17:49 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:17:50.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:17:50 smithi099 ceph-mon[24594]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]': finished 2024-09-17T00:17:50.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:17:50 smithi099 ceph-mon[24594]: osdmap e69: 8 total, 8 up, 8 in 2024-09-17T00:17:50.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:17:50 smithi099 ceph-mon[24594]: from='client.? 172.21.15.2:0/3113681048' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-09-17T00:17:50.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:17:50 smithi099 ceph-mon[24594]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-09-17T00:17:50.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:17:50 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:17:50.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:17:50 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:17:50.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:17:50 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:17:50.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:17:50 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:17:51.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:17:50 smithi002 ceph-mon[23737]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]': finished 2024-09-17T00:17:51.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:17:50 smithi002 ceph-mon[23737]: osdmap e69: 8 total, 8 up, 8 in 2024-09-17T00:17:51.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:17:50 smithi002 ceph-mon[23737]: from='client.? 172.21.15.2:0/3113681048' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-09-17T00:17:51.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:17:50 smithi002 ceph-mon[23737]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-09-17T00:17:51.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:17:50 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:17:51.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:17:50 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:17:51.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:17:50 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:17:51.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:17:50 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:17:51.230 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:17:50 smithi002 ceph-mon[28203]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]': finished 2024-09-17T00:17:51.230 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:17:50 smithi002 ceph-mon[28203]: osdmap e69: 8 total, 8 up, 8 in 2024-09-17T00:17:51.230 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:17:50 smithi002 ceph-mon[28203]: from='client.? 172.21.15.2:0/3113681048' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-09-17T00:17:51.230 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:17:50 smithi002 ceph-mon[28203]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-09-17T00:17:51.230 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:17:50 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:17:51.230 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:17:50 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:17:51.230 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:17:50 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:17:51.230 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:17:50 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:17:51.818 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-09-17T00:17:51.818 INFO:teuthology.orchestra.run.smithi002.stdout: "id": "13e94068-d26c-42fd-b814-ad052bb44d32", 2024-09-17T00:17:51.818 INFO:teuthology.orchestra.run.smithi002.stdout: "epoch": 1, 2024-09-17T00:17:51.818 INFO:teuthology.orchestra.run.smithi002.stdout: "predecessor_uuid": "a600141c-4d91-42fd-bee1-a35c2eecebf3", 2024-09-17T00:17:51.818 INFO:teuthology.orchestra.run.smithi002.stdout: "sync_status": [], 2024-09-17T00:17:51.818 INFO:teuthology.orchestra.run.smithi002.stdout: "period_map": { 2024-09-17T00:17:51.818 INFO:teuthology.orchestra.run.smithi002.stdout: "id": "13e94068-d26c-42fd-b814-ad052bb44d32", 2024-09-17T00:17:51.818 INFO:teuthology.orchestra.run.smithi002.stdout: "zonegroups": [ 2024-09-17T00:17:51.818 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-09-17T00:17:51.818 INFO:teuthology.orchestra.run.smithi002.stdout: "id": "91ff7d2e-24a7-4ed1-aedf-e28741f6a264", 2024-09-17T00:17:51.819 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "default", 2024-09-17T00:17:51.819 INFO:teuthology.orchestra.run.smithi002.stdout: "api_name": "default", 2024-09-17T00:17:51.819 INFO:teuthology.orchestra.run.smithi002.stdout: "is_master": "true", 2024-09-17T00:17:51.819 INFO:teuthology.orchestra.run.smithi002.stdout: "endpoints": [], 2024-09-17T00:17:51.819 INFO:teuthology.orchestra.run.smithi002.stdout: "hostnames": [], 2024-09-17T00:17:51.819 INFO:teuthology.orchestra.run.smithi002.stdout: "hostnames_s3website": [], 2024-09-17T00:17:51.819 INFO:teuthology.orchestra.run.smithi002.stdout: "master_zone": "cec51b2d-fa24-475f-91d4-d3238f0fb553", 2024-09-17T00:17:51.819 INFO:teuthology.orchestra.run.smithi002.stdout: "zones": [ 2024-09-17T00:17:51.819 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-09-17T00:17:51.819 INFO:teuthology.orchestra.run.smithi002.stdout: "id": "cec51b2d-fa24-475f-91d4-d3238f0fb553", 2024-09-17T00:17:51.819 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "z", 2024-09-17T00:17:51.819 INFO:teuthology.orchestra.run.smithi002.stdout: "endpoints": [], 2024-09-17T00:17:51.819 INFO:teuthology.orchestra.run.smithi002.stdout: "log_meta": "false", 2024-09-17T00:17:51.819 INFO:teuthology.orchestra.run.smithi002.stdout: "log_data": "false", 2024-09-17T00:17:51.820 INFO:teuthology.orchestra.run.smithi002.stdout: "bucket_index_max_shards": 11, 2024-09-17T00:17:51.820 INFO:teuthology.orchestra.run.smithi002.stdout: "read_only": "false", 2024-09-17T00:17:51.820 INFO:teuthology.orchestra.run.smithi002.stdout: "tier_type": "", 2024-09-17T00:17:51.820 INFO:teuthology.orchestra.run.smithi002.stdout: "sync_from_all": "true", 2024-09-17T00:17:51.820 INFO:teuthology.orchestra.run.smithi002.stdout: "sync_from": [], 2024-09-17T00:17:51.820 INFO:teuthology.orchestra.run.smithi002.stdout: "redirect_zone": "" 2024-09-17T00:17:51.820 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-09-17T00:17:51.820 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-09-17T00:17:51.820 INFO:teuthology.orchestra.run.smithi002.stdout: "placement_targets": [ 2024-09-17T00:17:51.820 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-09-17T00:17:51.820 INFO:teuthology.orchestra.run.smithi002.stdout: "name": "default-placement", 2024-09-17T00:17:51.820 INFO:teuthology.orchestra.run.smithi002.stdout: "tags": [], 2024-09-17T00:17:51.820 INFO:teuthology.orchestra.run.smithi002.stdout: "storage_classes": [ 2024-09-17T00:17:51.820 INFO:teuthology.orchestra.run.smithi002.stdout: "STANDARD" 2024-09-17T00:17:51.820 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2024-09-17T00:17:51.821 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-09-17T00:17:51.821 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-09-17T00:17:51.821 INFO:teuthology.orchestra.run.smithi002.stdout: "default_placement": "default-placement", 2024-09-17T00:17:51.821 INFO:teuthology.orchestra.run.smithi002.stdout: "realm_id": "014fa0b8-4536-4b8c-891b-d533f09e574f", 2024-09-17T00:17:51.821 INFO:teuthology.orchestra.run.smithi002.stdout: "sync_policy": { 2024-09-17T00:17:51.821 INFO:teuthology.orchestra.run.smithi002.stdout: "groups": [] 2024-09-17T00:17:51.821 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-09-17T00:17:51.821 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-09-17T00:17:51.821 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-09-17T00:17:51.821 INFO:teuthology.orchestra.run.smithi002.stdout: "short_zone_ids": [ 2024-09-17T00:17:51.821 INFO:teuthology.orchestra.run.smithi002.stdout: { 2024-09-17T00:17:51.821 INFO:teuthology.orchestra.run.smithi002.stdout: "key": "cec51b2d-fa24-475f-91d4-d3238f0fb553", 2024-09-17T00:17:51.821 INFO:teuthology.orchestra.run.smithi002.stdout: "val": 563381693 2024-09-17T00:17:51.822 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-09-17T00:17:51.822 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2024-09-17T00:17:51.822 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-09-17T00:17:51.822 INFO:teuthology.orchestra.run.smithi002.stdout: "master_zonegroup": "91ff7d2e-24a7-4ed1-aedf-e28741f6a264", 2024-09-17T00:17:51.822 INFO:teuthology.orchestra.run.smithi002.stdout: "master_zone": "cec51b2d-fa24-475f-91d4-d3238f0fb553", 2024-09-17T00:17:51.822 INFO:teuthology.orchestra.run.smithi002.stdout: "period_config": { 2024-09-17T00:17:51.822 INFO:teuthology.orchestra.run.smithi002.stdout: "bucket_quota": { 2024-09-17T00:17:51.822 INFO:teuthology.orchestra.run.smithi002.stdout: "enabled": false, 2024-09-17T00:17:51.822 INFO:teuthology.orchestra.run.smithi002.stdout: "check_on_raw": false, 2024-09-17T00:17:51.822 INFO:teuthology.orchestra.run.smithi002.stdout: "max_size": -1, 2024-09-17T00:17:51.822 INFO:teuthology.orchestra.run.smithi002.stdout: "max_size_kb": 0, 2024-09-17T00:17:51.822 INFO:teuthology.orchestra.run.smithi002.stdout: "max_objects": -1 2024-09-17T00:17:51.822 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-09-17T00:17:51.822 INFO:teuthology.orchestra.run.smithi002.stdout: "user_quota": { 2024-09-17T00:17:51.823 INFO:teuthology.orchestra.run.smithi002.stdout: "enabled": false, 2024-09-17T00:17:51.823 INFO:teuthology.orchestra.run.smithi002.stdout: "check_on_raw": false, 2024-09-17T00:17:51.823 INFO:teuthology.orchestra.run.smithi002.stdout: "max_size": -1, 2024-09-17T00:17:51.823 INFO:teuthology.orchestra.run.smithi002.stdout: "max_size_kb": 0, 2024-09-17T00:17:51.823 INFO:teuthology.orchestra.run.smithi002.stdout: "max_objects": -1 2024-09-17T00:17:51.823 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-09-17T00:17:51.823 INFO:teuthology.orchestra.run.smithi002.stdout: "user_ratelimit": { 2024-09-17T00:17:51.823 INFO:teuthology.orchestra.run.smithi002.stdout: "max_read_ops": 0, 2024-09-17T00:17:51.823 INFO:teuthology.orchestra.run.smithi002.stdout: "max_write_ops": 0, 2024-09-17T00:17:51.823 INFO:teuthology.orchestra.run.smithi002.stdout: "max_read_bytes": 0, 2024-09-17T00:17:51.823 INFO:teuthology.orchestra.run.smithi002.stdout: "max_write_bytes": 0, 2024-09-17T00:17:51.823 INFO:teuthology.orchestra.run.smithi002.stdout: "enabled": false 2024-09-17T00:17:51.823 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-09-17T00:17:51.823 INFO:teuthology.orchestra.run.smithi002.stdout: "bucket_ratelimit": { 2024-09-17T00:17:51.824 INFO:teuthology.orchestra.run.smithi002.stdout: "max_read_ops": 0, 2024-09-17T00:17:51.824 INFO:teuthology.orchestra.run.smithi002.stdout: "max_write_ops": 0, 2024-09-17T00:17:51.824 INFO:teuthology.orchestra.run.smithi002.stdout: "max_read_bytes": 0, 2024-09-17T00:17:51.824 INFO:teuthology.orchestra.run.smithi002.stdout: "max_write_bytes": 0, 2024-09-17T00:17:51.824 INFO:teuthology.orchestra.run.smithi002.stdout: "enabled": false 2024-09-17T00:17:51.824 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-09-17T00:17:51.824 INFO:teuthology.orchestra.run.smithi002.stdout: "anonymous_ratelimit": { 2024-09-17T00:17:51.824 INFO:teuthology.orchestra.run.smithi002.stdout: "max_read_ops": 0, 2024-09-17T00:17:51.824 INFO:teuthology.orchestra.run.smithi002.stdout: "max_write_ops": 0, 2024-09-17T00:17:51.824 INFO:teuthology.orchestra.run.smithi002.stdout: "max_read_bytes": 0, 2024-09-17T00:17:51.824 INFO:teuthology.orchestra.run.smithi002.stdout: "max_write_bytes": 0, 2024-09-17T00:17:51.824 INFO:teuthology.orchestra.run.smithi002.stdout: "enabled": false 2024-09-17T00:17:51.824 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-09-17T00:17:51.824 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-09-17T00:17:51.825 INFO:teuthology.orchestra.run.smithi002.stdout: "realm_id": "014fa0b8-4536-4b8c-891b-d533f09e574f", 2024-09-17T00:17:51.825 INFO:teuthology.orchestra.run.smithi002.stdout: "realm_name": "r", 2024-09-17T00:17:51.825 INFO:teuthology.orchestra.run.smithi002.stdout: "realm_epoch": 2 2024-09-17T00:17:51.825 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-09-17T00:17:52.171 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:17:51 smithi099 ceph-mon[24594]: pgmap v62: 129 pgs: 64 unknown, 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-17T00:17:52.171 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:17:51 smithi099 ceph-mon[24594]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-17T00:17:52.171 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:17:51 smithi099 ceph-mon[24594]: osdmap e70: 8 total, 8 up, 8 in 2024-09-17T00:17:52.171 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:17:51 smithi099 ceph-mon[24594]: from='client.? 172.21.15.2:0/3113681048' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-09-17T00:17:52.171 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:17:51 smithi099 ceph-mon[24594]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-09-17T00:17:52.212 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:17:51 smithi002 ceph-mon[28203]: pgmap v62: 129 pgs: 64 unknown, 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-17T00:17:52.212 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:17:51 smithi002 ceph-mon[28203]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-17T00:17:52.212 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:17:51 smithi002 ceph-mon[28203]: osdmap e70: 8 total, 8 up, 8 in 2024-09-17T00:17:52.212 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:17:51 smithi002 ceph-mon[28203]: from='client.? 172.21.15.2:0/3113681048' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-09-17T00:17:52.212 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:17:51 smithi002 ceph-mon[28203]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-09-17T00:17:52.212 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:17:51 smithi002 ceph-mon[23737]: pgmap v62: 129 pgs: 64 unknown, 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-17T00:17:52.212 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:17:51 smithi002 ceph-mon[23737]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-17T00:17:52.212 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:17:51 smithi002 ceph-mon[23737]: osdmap e70: 8 total, 8 up, 8 in 2024-09-17T00:17:52.212 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:17:51 smithi002 ceph-mon[23737]: from='client.? 172.21.15.2:0/3113681048' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-09-17T00:17:52.212 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:17:51 smithi002 ceph-mon[23737]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-09-17T00:17:52.729 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Sep 17 00:17:52 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-alertmanager-a[48044]: level=info ts=2024-09-17T00:17:52.469Z caller=cluster.go:688 component=cluster msg="gossip settled; proceeding" elapsed=10.003237888s 2024-09-17T00:17:52.777 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3aad27a2-7489-11ef-bceb-c7b262605968 -e sha1=c2ddf9721ef254645201cdbabd9c8f49c30a586f -- bash -c 'ceph orch apply rgw foo --realm r --zone z --placement=2 --port=8000' 2024-09-17T00:17:53.068 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:17:52 smithi099 ceph-mon[24594]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]': finished 2024-09-17T00:17:53.068 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:17:52 smithi099 ceph-mon[24594]: osdmap e71: 8 total, 8 up, 8 in 2024-09-17T00:17:53.093 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:17:52 smithi002 ceph-mon[28203]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]': finished 2024-09-17T00:17:53.093 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:17:52 smithi002 ceph-mon[28203]: osdmap e71: 8 total, 8 up, 8 in 2024-09-17T00:17:53.093 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:17:52 smithi002 ceph-mon[23737]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]': finished 2024-09-17T00:17:53.093 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:17:52 smithi002 ceph-mon[23737]: osdmap e71: 8 total, 8 up, 8 in 2024-09-17T00:17:54.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:17:53 smithi002 ceph-mon[23737]: pgmap v65: 129 pgs: 129 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 511 B/s wr, 1 op/s 2024-09-17T00:17:54.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:17:53 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:17:54.115 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:17:53 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:17:54.115 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:17:53 smithi002 ceph-mon[28203]: pgmap v65: 129 pgs: 129 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 511 B/s wr, 1 op/s 2024-09-17T00:17:54.116 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:17:53 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:17:54.116 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:17:53 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:17:54.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:17:53 smithi099 ceph-mon[24594]: pgmap v65: 129 pgs: 129 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 511 B/s wr, 1 op/s 2024-09-17T00:17:54.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:17:53 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:17:54.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:17:53 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:17:54.758 INFO:teuthology.orchestra.run.smithi002.stdout:Scheduled rgw.foo update... 2024-09-17T00:17:55.217 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:17:54 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[25772]: ::ffff:172.21.15.99 - - [17/Sep/2024:00:17:54] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-09-17T00:17:55.474 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3aad27a2-7489-11ef-bceb-c7b262605968 -e sha1=c2ddf9721ef254645201cdbabd9c8f49c30a586f -- bash -c 'ceph orch apply rgw smpl' 2024-09-17T00:17:56.020 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:17:55 smithi002 ceph-mon[28203]: pgmap v66: 129 pgs: 129 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 182 B/s rd, 364 B/s wr, 1 op/s 2024-09-17T00:17:56.020 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:17:55 smithi002 ceph-mon[28203]: from='client.14685 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo", "realm": "r", "zone": "z", "placement": "2", "port": 8000, "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:17:56.020 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:17:55 smithi002 ceph-mon[28203]: Saving service rgw.foo spec with placement count:2 2024-09-17T00:17:56.020 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:17:55 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:17:56.020 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:17:55 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:17:56.020 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:17:55 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:17:56.020 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:17:55 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:17:56.020 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:17:55 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:17:56.020 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:17:55 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:17:56.021 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:17:55 smithi002 ceph-mon[28203]: Saving service rgw.foo spec with placement count:2 2024-09-17T00:17:56.021 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:17:55 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:17:56.021 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:17:55 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi002.uwdhcs", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T00:17:56.021 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:17:55 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi002.uwdhcs", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-17T00:17:56.021 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:17:55 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:17:56.021 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:17:55 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:17:56.021 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:17:55 smithi002 ceph-mon[28203]: Deploying daemon rgw.foo.smithi002.uwdhcs on smithi002 2024-09-17T00:17:56.021 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:17:55 smithi002 ceph-mon[23737]: pgmap v66: 129 pgs: 129 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 182 B/s rd, 364 B/s wr, 1 op/s 2024-09-17T00:17:56.021 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:17:55 smithi002 ceph-mon[23737]: from='client.14685 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo", "realm": "r", "zone": "z", "placement": "2", "port": 8000, "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:17:56.021 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:17:55 smithi002 ceph-mon[23737]: Saving service rgw.foo spec with placement count:2 2024-09-17T00:17:56.021 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:17:55 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:17:56.022 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:17:55 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:17:56.022 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:17:55 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:17:56.022 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:17:55 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:17:56.022 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:17:55 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:17:56.022 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:17:55 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:17:56.022 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:17:55 smithi002 ceph-mon[23737]: Saving service rgw.foo spec with placement count:2 2024-09-17T00:17:56.022 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:17:55 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:17:56.022 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:17:55 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi002.uwdhcs", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T00:17:56.022 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:17:55 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi002.uwdhcs", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-17T00:17:56.022 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:17:55 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:17:56.022 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:17:55 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:17:56.022 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:17:55 smithi002 ceph-mon[23737]: Deploying daemon rgw.foo.smithi002.uwdhcs on smithi002 2024-09-17T00:17:56.130 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:17:55 smithi099 ceph-mon[24594]: pgmap v66: 129 pgs: 129 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 182 B/s rd, 364 B/s wr, 1 op/s 2024-09-17T00:17:56.130 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:17:55 smithi099 ceph-mon[24594]: from='client.14685 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo", "realm": "r", "zone": "z", "placement": "2", "port": 8000, "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:17:56.130 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:17:55 smithi099 ceph-mon[24594]: Saving service rgw.foo spec with placement count:2 2024-09-17T00:17:56.131 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:17:55 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:17:56.131 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:17:55 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:17:56.131 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:17:55 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:17:56.131 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:17:55 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:17:56.131 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:17:55 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:17:56.131 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:17:55 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:17:56.131 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:17:55 smithi099 ceph-mon[24594]: Saving service rgw.foo spec with placement count:2 2024-09-17T00:17:56.131 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:17:55 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:17:56.131 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:17:55 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi002.uwdhcs", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T00:17:56.131 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:17:55 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi002.uwdhcs", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-17T00:17:56.131 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:17:55 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:17:56.131 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:17:55 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:17:56.131 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:17:55 smithi099 ceph-mon[24594]: Deploying daemon rgw.foo.smithi002.uwdhcs on smithi002 2024-09-17T00:17:58.033 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:17:57 smithi002 ceph-mon[28203]: pgmap v67: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.4 KiB/s wr, 3 op/s 2024-09-17T00:17:58.033 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:17:57 smithi002 ceph-mon[23737]: pgmap v67: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.4 KiB/s wr, 3 op/s 2024-09-17T00:17:58.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:17:57 smithi099 ceph-mon[24594]: pgmap v67: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.4 KiB/s wr, 3 op/s 2024-09-17T00:17:58.840 INFO:teuthology.orchestra.run.smithi002.stdout:Scheduled rgw.smpl update... 2024-09-17T00:17:59.479 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:17:59 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: ::ffff:172.21.15.99 - - [17/Sep/2024:00:17:59] "GET /metrics HTTP/1.1" 200 196795 "" "Prometheus/2.33.4" 2024-09-17T00:17:59.841 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3aad27a2-7489-11ef-bceb-c7b262605968 -e sha1=c2ddf9721ef254645201cdbabd9c8f49c30a586f -- bash -c 'ceph osd pool create foo' 2024-09-17T00:18:00.209 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:17:59 smithi002 ceph-mon[28203]: pgmap v68: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1.1 KiB/s wr, 2 op/s 2024-09-17T00:18:00.209 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:17:59 smithi002 ceph-mon[28203]: from='client.14691 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "smpl", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:18:00.209 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:17:59 smithi002 ceph-mon[28203]: Saving service rgw.smpl spec with placement count:2 2024-09-17T00:18:00.210 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:17:59 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:18:00.210 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:17:59 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:18:00.210 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:17:59 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi099.utzfap", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T00:18:00.210 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:17:59 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi099.utzfap", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-17T00:18:00.210 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:17:59 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:18:00.210 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:17:59 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:18:00.210 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:17:59 smithi002 ceph-mon[23737]: pgmap v68: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1.1 KiB/s wr, 2 op/s 2024-09-17T00:18:00.210 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:17:59 smithi002 ceph-mon[23737]: from='client.14691 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "smpl", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:18:00.210 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:17:59 smithi002 ceph-mon[23737]: Saving service rgw.smpl spec with placement count:2 2024-09-17T00:18:00.210 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:17:59 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:18:00.210 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:17:59 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:18:00.211 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:17:59 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi099.utzfap", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T00:18:00.211 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:17:59 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi099.utzfap", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-17T00:18:00.211 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:17:59 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:18:00.211 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:17:59 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:18:00.214 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:17:59 smithi099 ceph-mon[24594]: pgmap v68: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1.1 KiB/s wr, 2 op/s 2024-09-17T00:18:00.214 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:17:59 smithi099 ceph-mon[24594]: from='client.14691 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "smpl", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:18:00.214 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:17:59 smithi099 ceph-mon[24594]: Saving service rgw.smpl spec with placement count:2 2024-09-17T00:18:00.215 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:17:59 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:18:00.215 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:17:59 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:18:00.215 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:17:59 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi099.utzfap", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T00:18:00.215 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:17:59 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi099.utzfap", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-17T00:18:00.215 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:17:59 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:18:00.215 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:17:59 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:18:01.196 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:00 smithi099 ceph-mon[24594]: Deploying daemon rgw.foo.smithi099.utzfap on smithi099 2024-09-17T00:18:01.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:00 smithi002 ceph-mon[23737]: Deploying daemon rgw.foo.smithi099.utzfap on smithi099 2024-09-17T00:18:01.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:00 smithi002 ceph-mon[28203]: Deploying daemon rgw.foo.smithi099.utzfap on smithi099 2024-09-17T00:18:02.159 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:01 smithi099 ceph-mon[24594]: pgmap v69: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 958 B/s wr, 2 op/s 2024-09-17T00:18:02.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:01 smithi002 ceph-mon[28203]: pgmap v69: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 958 B/s wr, 2 op/s 2024-09-17T00:18:02.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:01 smithi002 ceph-mon[23737]: pgmap v69: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 958 B/s wr, 2 op/s 2024-09-17T00:18:02.862 INFO:teuthology.orchestra.run.smithi002.stderr:pool 'foo' created 2024-09-17T00:18:02.966 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:02 smithi099 ceph-mon[24594]: from='client.? 172.21.15.2:0/3151763156' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-09-17T00:18:02.966 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:02 smithi099 ceph-mon[24594]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-09-17T00:18:02.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:02 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:18:02.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:02 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:18:02.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:02 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:18:02.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:02 smithi002 ceph-mon[28203]: from='client.? 172.21.15.2:0/3151763156' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-09-17T00:18:02.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:02 smithi002 ceph-mon[28203]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-09-17T00:18:02.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:02 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:18:02.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:02 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:18:02.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:02 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:18:02.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:02 smithi002 ceph-mon[23737]: from='client.? 172.21.15.2:0/3151763156' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-09-17T00:18:02.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:02 smithi002 ceph-mon[23737]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-09-17T00:18:02.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:02 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:18:02.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:02 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:18:02.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:02 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:18:03.776 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3aad27a2-7489-11ef-bceb-c7b262605968 -e sha1=c2ddf9721ef254645201cdbabd9c8f49c30a586f -- bash -c 'rbd pool init foo' 2024-09-17T00:18:03.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:03 smithi099 ceph-mon[24594]: pgmap v70: 129 pgs: 129 active+clean; 584 KiB data, 54 MiB used, 715 GiB / 715 GiB avail; 76 KiB/s rd, 3.8 KiB/s wr, 159 op/s 2024-09-17T00:18:03.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:03 smithi099 ceph-mon[24594]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "foo"}]': finished 2024-09-17T00:18:03.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:03 smithi099 ceph-mon[24594]: osdmap e72: 8 total, 8 up, 8 in 2024-09-17T00:18:04.042 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:03 smithi002 ceph-mon[28203]: pgmap v70: 129 pgs: 129 active+clean; 584 KiB data, 54 MiB used, 715 GiB / 715 GiB avail; 76 KiB/s rd, 3.8 KiB/s wr, 159 op/s 2024-09-17T00:18:04.042 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:03 smithi002 ceph-mon[28203]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "foo"}]': finished 2024-09-17T00:18:04.042 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:03 smithi002 ceph-mon[28203]: osdmap e72: 8 total, 8 up, 8 in 2024-09-17T00:18:04.042 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:03 smithi002 ceph-mon[23737]: pgmap v70: 129 pgs: 129 active+clean; 584 KiB data, 54 MiB used, 715 GiB / 715 GiB avail; 76 KiB/s rd, 3.8 KiB/s wr, 159 op/s 2024-09-17T00:18:04.042 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:03 smithi002 ceph-mon[23737]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "foo"}]': finished 2024-09-17T00:18:04.042 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:03 smithi002 ceph-mon[23737]: osdmap e72: 8 total, 8 up, 8 in 2024-09-17T00:18:05.216 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:04 smithi099 ceph-mon[24594]: osdmap e73: 8 total, 8 up, 8 in 2024-09-17T00:18:05.217 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:18:04 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[25772]: ::ffff:172.21.15.99 - - [17/Sep/2024:00:18:04] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-09-17T00:18:05.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:04 smithi002 ceph-mon[23737]: osdmap e73: 8 total, 8 up, 8 in 2024-09-17T00:18:05.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:04 smithi002 ceph-mon[28203]: osdmap e73: 8 total, 8 up, 8 in 2024-09-17T00:18:06.179 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:05 smithi002 ceph-mon[28203]: pgmap v73: 161 pgs: 32 unknown, 129 active+clean; 584 KiB data, 54 MiB used, 715 GiB / 715 GiB avail; 100 KiB/s rd, 4.1 KiB/s wr, 209 op/s 2024-09-17T00:18:06.179 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:05 smithi002 ceph-mon[28203]: osdmap e74: 8 total, 8 up, 8 in 2024-09-17T00:18:06.179 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:05 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:18:06.179 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:05 smithi002 ceph-mon[23737]: pgmap v73: 161 pgs: 32 unknown, 129 active+clean; 584 KiB data, 54 MiB used, 715 GiB / 715 GiB avail; 100 KiB/s rd, 4.1 KiB/s wr, 209 op/s 2024-09-17T00:18:06.179 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:05 smithi002 ceph-mon[23737]: osdmap e74: 8 total, 8 up, 8 in 2024-09-17T00:18:06.179 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:05 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:18:06.216 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:05 smithi099 ceph-mon[24594]: pgmap v73: 161 pgs: 32 unknown, 129 active+clean; 584 KiB data, 54 MiB used, 715 GiB / 715 GiB avail; 100 KiB/s rd, 4.1 KiB/s wr, 209 op/s 2024-09-17T00:18:06.216 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:05 smithi099 ceph-mon[24594]: osdmap e74: 8 total, 8 up, 8 in 2024-09-17T00:18:06.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:05 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:18:07.216 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:06 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:18:07.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:06 smithi099 ceph-mon[24594]: from='client.? 172.21.15.2:0/1342307965' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-09-17T00:18:07.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:06 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:18:07.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:06 smithi002 ceph-mon[28203]: from='client.? 172.21.15.2:0/1342307965' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-09-17T00:18:07.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:06 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:18:07.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:06 smithi002 ceph-mon[23737]: from='client.? 172.21.15.2:0/1342307965' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-09-17T00:18:08.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:07 smithi099 ceph-mon[24594]: pgmap v75: 161 pgs: 161 active+clean; 584 KiB data, 64 MiB used, 715 GiB / 715 GiB avail; 283 KiB/s rd, 5.7 KiB/s wr, 508 op/s 2024-09-17T00:18:08.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:07 smithi099 ceph-mon[24594]: from='client.? 172.21.15.2:0/1342307965' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]': finished 2024-09-17T00:18:08.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:07 smithi099 ceph-mon[24594]: osdmap e75: 8 total, 8 up, 8 in 2024-09-17T00:18:08.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:07 smithi002 ceph-mon[23737]: pgmap v75: 161 pgs: 161 active+clean; 584 KiB data, 64 MiB used, 715 GiB / 715 GiB avail; 283 KiB/s rd, 5.7 KiB/s wr, 508 op/s 2024-09-17T00:18:08.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:07 smithi002 ceph-mon[23737]: from='client.? 172.21.15.2:0/1342307965' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]': finished 2024-09-17T00:18:08.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:07 smithi002 ceph-mon[23737]: osdmap e75: 8 total, 8 up, 8 in 2024-09-17T00:18:08.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:07 smithi002 ceph-mon[28203]: pgmap v75: 161 pgs: 161 active+clean; 584 KiB data, 64 MiB used, 715 GiB / 715 GiB avail; 283 KiB/s rd, 5.7 KiB/s wr, 508 op/s 2024-09-17T00:18:08.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:07 smithi002 ceph-mon[28203]: from='client.? 172.21.15.2:0/1342307965' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]': finished 2024-09-17T00:18:08.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:07 smithi002 ceph-mon[28203]: osdmap e75: 8 total, 8 up, 8 in 2024-09-17T00:18:08.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:08 smithi002 ceph-mon[28203]: osdmap e76: 8 total, 8 up, 8 in 2024-09-17T00:18:08.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:08 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:18:08.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:08 smithi002 ceph-mon[28203]: Checking dashboard <-> RGW credentials 2024-09-17T00:18:08.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:08 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:18:08.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:08 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:18:08.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:08 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:18:08.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:08 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:18:08.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:08 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi002.rqnxii", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T00:18:08.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:08 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi002.rqnxii", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-17T00:18:08.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:08 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:18:08.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:08 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:18:08.981 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:08 smithi002 ceph-mon[23737]: osdmap e76: 8 total, 8 up, 8 in 2024-09-17T00:18:08.981 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:08 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:18:08.981 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:08 smithi002 ceph-mon[23737]: Checking dashboard <-> RGW credentials 2024-09-17T00:18:08.981 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:08 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:18:08.981 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:08 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:18:08.981 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:08 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:18:08.981 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:08 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:18:08.981 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:08 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi002.rqnxii", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T00:18:08.981 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:08 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi002.rqnxii", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-17T00:18:08.981 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:08 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:18:08.982 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:08 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:18:09.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:08 smithi099 ceph-mon[24594]: osdmap e76: 8 total, 8 up, 8 in 2024-09-17T00:18:09.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:08 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:18:09.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:08 smithi099 ceph-mon[24594]: Checking dashboard <-> RGW credentials 2024-09-17T00:18:09.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:08 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:18:09.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:08 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:18:09.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:08 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:18:09.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:08 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:18:09.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:08 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi002.rqnxii", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T00:18:09.218 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:08 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi002.rqnxii", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-17T00:18:09.218 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:08 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:18:09.218 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:08 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:18:09.338 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:18:09 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: ::ffff:172.21.15.99 - - [17/Sep/2024:00:18:09] "GET /metrics HTTP/1.1" 200 196795 "" "Prometheus/2.33.4" 2024-09-17T00:18:09.986 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:09 smithi002 ceph-mon[23737]: Saving service rgw.smpl spec with placement count:2 2024-09-17T00:18:09.987 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:09 smithi002 ceph-mon[23737]: Deploying daemon rgw.smpl.smithi002.rqnxii on smithi002 2024-09-17T00:18:09.987 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:09 smithi002 ceph-mon[23737]: pgmap v78: 161 pgs: 161 active+clean; 584 KiB data, 64 MiB used, 715 GiB / 715 GiB avail; 199 KiB/s rd, 226 B/s wr, 305 op/s 2024-09-17T00:18:09.987 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:09 smithi002 ceph-mon[23737]: osdmap e77: 8 total, 8 up, 8 in 2024-09-17T00:18:09.987 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:09 smithi002 ceph-mon[28203]: Saving service rgw.smpl spec with placement count:2 2024-09-17T00:18:09.987 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:09 smithi002 ceph-mon[28203]: Deploying daemon rgw.smpl.smithi002.rqnxii on smithi002 2024-09-17T00:18:09.987 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:09 smithi002 ceph-mon[28203]: pgmap v78: 161 pgs: 161 active+clean; 584 KiB data, 64 MiB used, 715 GiB / 715 GiB avail; 199 KiB/s rd, 226 B/s wr, 305 op/s 2024-09-17T00:18:09.987 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:09 smithi002 ceph-mon[28203]: osdmap e77: 8 total, 8 up, 8 in 2024-09-17T00:18:09.988 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3aad27a2-7489-11ef-bceb-c7b262605968 -e sha1=c2ddf9721ef254645201cdbabd9c8f49c30a586f -- bash -c 'ceph orch apply iscsi foo u p' 2024-09-17T00:18:10.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:09 smithi099 ceph-mon[24594]: Saving service rgw.smpl spec with placement count:2 2024-09-17T00:18:10.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:09 smithi099 ceph-mon[24594]: Deploying daemon rgw.smpl.smithi002.rqnxii on smithi002 2024-09-17T00:18:10.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:09 smithi099 ceph-mon[24594]: pgmap v78: 161 pgs: 161 active+clean; 584 KiB data, 64 MiB used, 715 GiB / 715 GiB avail; 199 KiB/s rd, 226 B/s wr, 305 op/s 2024-09-17T00:18:10.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:09 smithi099 ceph-mon[24594]: osdmap e77: 8 total, 8 up, 8 in 2024-09-17T00:18:11.216 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:10 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.12", "id": [7, 2]}]: dispatch 2024-09-17T00:18:11.216 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:10 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]: dispatch 2024-09-17T00:18:11.216 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:10 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]: dispatch 2024-09-17T00:18:11.216 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:10 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]}]: dispatch 2024-09-17T00:18:11.216 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:10 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T00:18:11.216 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:10 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T00:18:11.216 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:10 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.12", "id": [7, 2]}]: dispatch 2024-09-17T00:18:11.217 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:10 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]: dispatch 2024-09-17T00:18:11.217 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:10 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]: dispatch 2024-09-17T00:18:11.217 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:10 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]}]: dispatch 2024-09-17T00:18:11.217 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:10 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T00:18:11.217 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:10 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T00:18:11.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:10 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.12", "id": [7, 2]}]: dispatch 2024-09-17T00:18:11.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:10 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]: dispatch 2024-09-17T00:18:11.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:10 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]: dispatch 2024-09-17T00:18:11.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:10 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]}]: dispatch 2024-09-17T00:18:11.218 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:10 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T00:18:11.218 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:10 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T00:18:12.131 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:11 smithi002 ceph-mon[28203]: pgmap v80: 161 pgs: 161 active+clean; 584 KiB data, 64 MiB used, 715 GiB / 715 GiB avail; 163 KiB/s rd, 185 B/s wr, 250 op/s 2024-09-17T00:18:12.131 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:11 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.12", "id": [7, 2]}]': finished 2024-09-17T00:18:12.132 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:11 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]': finished 2024-09-17T00:18:12.132 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:11 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]': finished 2024-09-17T00:18:12.132 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:11 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]}]': finished 2024-09-17T00:18:12.132 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:11 smithi002 ceph-mon[28203]: osdmap e78: 8 total, 8 up, 8 in 2024-09-17T00:18:12.132 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:11 smithi002 ceph-mon[23737]: pgmap v80: 161 pgs: 161 active+clean; 584 KiB data, 64 MiB used, 715 GiB / 715 GiB avail; 163 KiB/s rd, 185 B/s wr, 250 op/s 2024-09-17T00:18:12.132 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:11 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.12", "id": [7, 2]}]': finished 2024-09-17T00:18:12.132 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:11 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]': finished 2024-09-17T00:18:12.132 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:11 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]': finished 2024-09-17T00:18:12.132 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:11 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]}]': finished 2024-09-17T00:18:12.132 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:11 smithi002 ceph-mon[23737]: osdmap e78: 8 total, 8 up, 8 in 2024-09-17T00:18:12.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:11 smithi099 ceph-mon[24594]: pgmap v80: 161 pgs: 161 active+clean; 584 KiB data, 64 MiB used, 715 GiB / 715 GiB avail; 163 KiB/s rd, 185 B/s wr, 250 op/s 2024-09-17T00:18:12.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:11 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.12", "id": [7, 2]}]': finished 2024-09-17T00:18:12.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:11 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.4", "id": [1, 5]}]': finished 2024-09-17T00:18:12.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:11 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]': finished 2024-09-17T00:18:12.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:11 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 2]}]': finished 2024-09-17T00:18:12.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:11 smithi099 ceph-mon[24594]: osdmap e78: 8 total, 8 up, 8 in 2024-09-17T00:18:12.603 INFO:teuthology.orchestra.run.smithi002.stdout:Scheduled iscsi.foo update... 2024-09-17T00:18:13.173 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:12 smithi002 ceph-mon[28203]: osdmap e79: 8 total, 8 up, 8 in 2024-09-17T00:18:13.173 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:12 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:18:13.173 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:12 smithi002 ceph-mon[23737]: osdmap e79: 8 total, 8 up, 8 in 2024-09-17T00:18:13.173 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:12 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:18:13.216 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:12 smithi099 ceph-mon[24594]: osdmap e79: 8 total, 8 up, 8 in 2024-09-17T00:18:13.216 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:12 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:18:13.585 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3aad27a2-7489-11ef-bceb-c7b262605968 -e sha1=c2ddf9721ef254645201cdbabd9c8f49c30a586f -- bash -c 'sleep 120' 2024-09-17T00:18:13.894 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Sep 17 00:18:13 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-alertmanager-a[48044]: level=warn ts=2024-09-17T00:18:13.522Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.2:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.2 because it doesn't contain any IP SANs" 2024-09-17T00:18:13.894 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Sep 17 00:18:13 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-alertmanager-a[48044]: level=warn ts=2024-09-17T00:18:13.522Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.99:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.99 because it doesn't contain any IP SANs" 2024-09-17T00:18:14.216 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:14 smithi099 ceph-mon[24594]: pgmap v83: 161 pgs: 4 peering, 157 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 915 B/s wr, 35 op/s 2024-09-17T00:18:14.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:14 smithi099 ceph-mon[24594]: from='client.24685 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "foo", "api_user": "u", "api_password": "p", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:18:14.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:14 smithi099 ceph-mon[24594]: Saving service iscsi.foo spec with placement count:1 2024-09-17T00:18:14.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:14 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:18:14.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:14 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi099.nzvkpm", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T00:18:14.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:14 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi099.nzvkpm", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-17T00:18:14.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:14 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:18:14.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:14 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:18:14.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:14 smithi099 ceph-mon[24594]: Deploying daemon rgw.smpl.smithi099.nzvkpm on smithi099 2024-09-17T00:18:14.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:14 smithi002 ceph-mon[28203]: pgmap v83: 161 pgs: 4 peering, 157 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 915 B/s wr, 35 op/s 2024-09-17T00:18:14.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:14 smithi002 ceph-mon[28203]: from='client.24685 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "foo", "api_user": "u", "api_password": "p", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:18:14.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:14 smithi002 ceph-mon[28203]: Saving service iscsi.foo spec with placement count:1 2024-09-17T00:18:14.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:14 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:18:14.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:14 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi099.nzvkpm", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T00:18:14.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:14 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi099.nzvkpm", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-17T00:18:14.480 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:14 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:18:14.480 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:14 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:18:14.480 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:14 smithi002 ceph-mon[28203]: Deploying daemon rgw.smpl.smithi099.nzvkpm on smithi099 2024-09-17T00:18:14.480 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:14 smithi002 ceph-mon[23737]: pgmap v83: 161 pgs: 4 peering, 157 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 26 KiB/s rd, 915 B/s wr, 35 op/s 2024-09-17T00:18:14.480 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:14 smithi002 ceph-mon[23737]: from='client.24685 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "foo", "api_user": "u", "api_password": "p", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:18:14.480 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:14 smithi002 ceph-mon[23737]: Saving service iscsi.foo spec with placement count:1 2024-09-17T00:18:14.480 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:14 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:18:14.480 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:14 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi099.nzvkpm", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T00:18:14.480 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:14 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi099.nzvkpm", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-17T00:18:14.480 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:14 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:18:14.480 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:14 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:18:14.481 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:14 smithi002 ceph-mon[23737]: Deploying daemon rgw.smpl.smithi099.nzvkpm on smithi099 2024-09-17T00:18:15.216 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:18:14 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[25772]: ::ffff:172.21.15.99 - - [17/Sep/2024:00:18:14] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-09-17T00:18:16.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:16 smithi002 ceph-mon[28203]: pgmap v84: 161 pgs: 4 peering, 157 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 682 B/s wr, 26 op/s 2024-09-17T00:18:16.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:16 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:18:16.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:16 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:18:16.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:16 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:18:16.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:16 smithi002 ceph-mon[23737]: pgmap v84: 161 pgs: 4 peering, 157 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 682 B/s wr, 26 op/s 2024-09-17T00:18:16.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:16 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:18:16.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:16 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:18:16.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:16 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:18:16.466 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:16 smithi099 ceph-mon[24594]: pgmap v84: 161 pgs: 4 peering, 157 active+clean; 585 KiB data, 69 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 682 B/s wr, 26 op/s 2024-09-17T00:18:16.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:16 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:18:16.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:16 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:18:16.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:16 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:18:18.466 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:18 smithi099 ceph-mon[24594]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 300 KiB/s rd, 822 B/s wr, 455 op/s; 6 B/s, 0 objects/s recovering 2024-09-17T00:18:18.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:18 smithi099 ceph-mon[24594]: osdmap e80: 8 total, 8 up, 8 in 2024-09-17T00:18:18.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:18 smithi002 ceph-mon[23737]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 300 KiB/s rd, 822 B/s wr, 455 op/s; 6 B/s, 0 objects/s recovering 2024-09-17T00:18:18.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:18 smithi002 ceph-mon[23737]: osdmap e80: 8 total, 8 up, 8 in 2024-09-17T00:18:18.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:18 smithi002 ceph-mon[28203]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 300 KiB/s rd, 822 B/s wr, 455 op/s; 6 B/s, 0 objects/s recovering 2024-09-17T00:18:18.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:18 smithi002 ceph-mon[28203]: osdmap e80: 8 total, 8 up, 8 in 2024-09-17T00:18:19.446 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:18:19 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: ::ffff:172.21.15.99 - - [17/Sep/2024:00:18:19] "GET /metrics HTTP/1.1" 200 207951 "" "Prometheus/2.33.4" 2024-09-17T00:18:19.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:19 smithi002 ceph-mon[23737]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 300 KiB/s rd, 823 B/s wr, 456 op/s; 6 B/s, 0 objects/s recovering 2024-09-17T00:18:19.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:19 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:18:19.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:19 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:18:19.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:19 smithi002 ceph-mon[23737]: Checking dashboard <-> RGW credentials 2024-09-17T00:18:19.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:19 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:18:19.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:19 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi002.vmouem", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-17T00:18:19.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:19 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi002.vmouem", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-09-17T00:18:19.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:19 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:18:19.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:19 smithi002 ceph-mon[28203]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 300 KiB/s rd, 823 B/s wr, 456 op/s; 6 B/s, 0 objects/s recovering 2024-09-17T00:18:19.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:19 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:18:19.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:19 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:18:19.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:19 smithi002 ceph-mon[28203]: Checking dashboard <-> RGW credentials 2024-09-17T00:18:19.731 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:19 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:18:19.731 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:19 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi002.vmouem", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-17T00:18:19.731 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:19 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi002.vmouem", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-09-17T00:18:19.731 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:19 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:18:19.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:19 smithi099 ceph-mon[24594]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 300 KiB/s rd, 823 B/s wr, 456 op/s; 6 B/s, 0 objects/s recovering 2024-09-17T00:18:19.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:19 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:18:19.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:19 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:18:19.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:19 smithi099 ceph-mon[24594]: Checking dashboard <-> RGW credentials 2024-09-17T00:18:19.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:19 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:18:19.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:19 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi002.vmouem", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-17T00:18:19.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:19 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi002.vmouem", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-09-17T00:18:19.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:19 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:18:20.966 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:20 smithi099 ceph-mon[24594]: Checking pool "foo" exists for service iscsi.foo 2024-09-17T00:18:20.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:20 smithi099 ceph-mon[24594]: Deploying daemon iscsi.foo.smithi002.vmouem on smithi002 2024-09-17T00:18:20.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:20 smithi099 ceph-mon[24594]: mgrmap e21: y(active, since 2m), standbys: x 2024-09-17T00:18:20.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:20 smithi002 ceph-mon[23737]: Checking pool "foo" exists for service iscsi.foo 2024-09-17T00:18:20.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:20 smithi002 ceph-mon[23737]: Deploying daemon iscsi.foo.smithi002.vmouem on smithi002 2024-09-17T00:18:20.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:20 smithi002 ceph-mon[23737]: mgrmap e21: y(active, since 2m), standbys: x 2024-09-17T00:18:20.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:20 smithi002 ceph-mon[28203]: Checking pool "foo" exists for service iscsi.foo 2024-09-17T00:18:20.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:20 smithi002 ceph-mon[28203]: Deploying daemon iscsi.foo.smithi002.vmouem on smithi002 2024-09-17T00:18:20.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:20 smithi002 ceph-mon[28203]: mgrmap e21: y(active, since 2m), standbys: x 2024-09-17T00:18:22.178 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:21 smithi002 ceph-mon[23737]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 251 KiB/s rd, 242 B/s wr, 383 op/s; 5 B/s, 0 objects/s recovering 2024-09-17T00:18:22.178 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:21 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:18:22.178 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:21 smithi002 ceph-mon[28203]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 251 KiB/s rd, 242 B/s wr, 383 op/s; 5 B/s, 0 objects/s recovering 2024-09-17T00:18:22.179 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:21 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:18:22.216 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:21 smithi099 ceph-mon[24594]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 251 KiB/s rd, 242 B/s wr, 383 op/s; 5 B/s, 0 objects/s recovering 2024-09-17T00:18:22.216 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:21 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:18:23.823 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Sep 17 00:18:23 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-alertmanager-a[48044]: level=error ts=2024-09-17T00:18:23.509Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.99:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.99 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.2:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.2 because it doesn't contain any IP SANs" 2024-09-17T00:18:23.823 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Sep 17 00:18:23 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-alertmanager-a[48044]: level=warn ts=2024-09-17T00:18:23.511Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.2:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.2 because it doesn't contain any IP SANs" 2024-09-17T00:18:23.823 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Sep 17 00:18:23 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-alertmanager-a[48044]: level=warn ts=2024-09-17T00:18:23.512Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.99:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.99 because it doesn't contain any IP SANs" 2024-09-17T00:18:23.824 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:23 smithi002 ceph-mon[28203]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 217 KiB/s rd, 204 B/s wr, 330 op/s; 4 B/s, 0 objects/s recovering 2024-09-17T00:18:23.824 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:23 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:18:23.824 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:23 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:18:23.824 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:23 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:18:23.824 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:23 smithi002 ceph-mon[28203]: from='client.? 172.21.15.2:0/2586268062' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-17T00:18:23.824 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:23 smithi002 ceph-mon[28203]: from='client.? 172.21.15.2:0/401541708' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/1056079102"}]: dispatch 2024-09-17T00:18:23.824 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:23 smithi002 ceph-mon[28203]: from='client.? ' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/1056079102"}]: dispatch 2024-09-17T00:18:23.824 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:23 smithi002 ceph-mon[23737]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 217 KiB/s rd, 204 B/s wr, 330 op/s; 4 B/s, 0 objects/s recovering 2024-09-17T00:18:23.824 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:23 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:18:23.824 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:23 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:18:23.824 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:23 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:18:23.825 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:23 smithi002 ceph-mon[23737]: from='client.? 172.21.15.2:0/2586268062' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-17T00:18:23.825 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:23 smithi002 ceph-mon[23737]: from='client.? 172.21.15.2:0/401541708' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/1056079102"}]: dispatch 2024-09-17T00:18:23.825 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:23 smithi002 ceph-mon[23737]: from='client.? ' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/1056079102"}]: dispatch 2024-09-17T00:18:23.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:23 smithi099 ceph-mon[24594]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 217 KiB/s rd, 204 B/s wr, 330 op/s; 4 B/s, 0 objects/s recovering 2024-09-17T00:18:23.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:23 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:18:23.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:23 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:18:23.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:23 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:18:23.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:23 smithi099 ceph-mon[24594]: from='client.? 172.21.15.2:0/2586268062' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-17T00:18:23.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:23 smithi099 ceph-mon[24594]: from='client.? 172.21.15.2:0/401541708' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/1056079102"}]: dispatch 2024-09-17T00:18:23.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:23 smithi099 ceph-mon[24594]: from='client.? ' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/1056079102"}]: dispatch 2024-09-17T00:18:25.075 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:24 smithi002 ceph-mon[28203]: from='client.? ' entity='client.iscsi.foo.smithi002.vmouem' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/1056079102"}]': finished 2024-09-17T00:18:25.075 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:24 smithi002 ceph-mon[28203]: osdmap e81: 8 total, 8 up, 8 in 2024-09-17T00:18:25.075 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:24 smithi002 ceph-mon[28203]: from='client.? 172.21.15.2:0/3490816190' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6801/3807888742"}]: dispatch 2024-09-17T00:18:25.075 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:24 smithi002 ceph-mon[28203]: from='client.? ' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6801/3807888742"}]: dispatch 2024-09-17T00:18:25.075 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:24 smithi002 ceph-mon[23737]: from='client.? ' entity='client.iscsi.foo.smithi002.vmouem' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/1056079102"}]': finished 2024-09-17T00:18:25.075 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:24 smithi002 ceph-mon[23737]: osdmap e81: 8 total, 8 up, 8 in 2024-09-17T00:18:25.076 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:24 smithi002 ceph-mon[23737]: from='client.? 172.21.15.2:0/3490816190' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6801/3807888742"}]: dispatch 2024-09-17T00:18:25.076 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:24 smithi002 ceph-mon[23737]: from='client.? ' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6801/3807888742"}]: dispatch 2024-09-17T00:18:25.216 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:24 smithi099 ceph-mon[24594]: from='client.? ' entity='client.iscsi.foo.smithi002.vmouem' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/1056079102"}]': finished 2024-09-17T00:18:25.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:24 smithi099 ceph-mon[24594]: osdmap e81: 8 total, 8 up, 8 in 2024-09-17T00:18:25.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:24 smithi099 ceph-mon[24594]: from='client.? 172.21.15.2:0/3490816190' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6801/3807888742"}]: dispatch 2024-09-17T00:18:25.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:24 smithi099 ceph-mon[24594]: from='client.? ' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6801/3807888742"}]: dispatch 2024-09-17T00:18:25.217 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:18:24 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[25772]: ::ffff:172.21.15.99 - - [17/Sep/2024:00:18:24] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-09-17T00:18:26.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:25 smithi099 ceph-mon[24594]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 6.5 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-17T00:18:26.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:25 smithi099 ceph-mon[24594]: from='client.? ' entity='client.iscsi.foo.smithi002.vmouem' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6801/3807888742"}]': finished 2024-09-17T00:18:26.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:25 smithi099 ceph-mon[24594]: osdmap e82: 8 total, 8 up, 8 in 2024-09-17T00:18:26.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:25 smithi099 ceph-mon[24594]: from='client.? 172.21.15.2:0/1372570122' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6800/3807888742"}]: dispatch 2024-09-17T00:18:26.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:25 smithi099 ceph-mon[24594]: from='client.? ' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6800/3807888742"}]: dispatch 2024-09-17T00:18:26.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:25 smithi002 ceph-mon[23737]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 6.5 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-17T00:18:26.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:25 smithi002 ceph-mon[23737]: from='client.? ' entity='client.iscsi.foo.smithi002.vmouem' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6801/3807888742"}]': finished 2024-09-17T00:18:26.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:25 smithi002 ceph-mon[23737]: osdmap e82: 8 total, 8 up, 8 in 2024-09-17T00:18:26.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:25 smithi002 ceph-mon[23737]: from='client.? 172.21.15.2:0/1372570122' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6800/3807888742"}]: dispatch 2024-09-17T00:18:26.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:25 smithi002 ceph-mon[23737]: from='client.? ' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6800/3807888742"}]: dispatch 2024-09-17T00:18:26.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:25 smithi002 ceph-mon[28203]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 6.5 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-17T00:18:26.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:25 smithi002 ceph-mon[28203]: from='client.? ' entity='client.iscsi.foo.smithi002.vmouem' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6801/3807888742"}]': finished 2024-09-17T00:18:26.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:25 smithi002 ceph-mon[28203]: osdmap e82: 8 total, 8 up, 8 in 2024-09-17T00:18:26.230 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:25 smithi002 ceph-mon[28203]: from='client.? 172.21.15.2:0/1372570122' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6800/3807888742"}]: dispatch 2024-09-17T00:18:26.230 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:25 smithi002 ceph-mon[28203]: from='client.? ' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6800/3807888742"}]: dispatch 2024-09-17T00:18:26.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:26 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:18:26.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:26 smithi099 ceph-mon[24594]: from='client.? ' entity='client.iscsi.foo.smithi002.vmouem' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6800/3807888742"}]': finished 2024-09-17T00:18:26.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:26 smithi099 ceph-mon[24594]: osdmap e83: 8 total, 8 up, 8 in 2024-09-17T00:18:26.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:26 smithi099 ceph-mon[24594]: from='client.? 172.21.15.2:0/312247546' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/1754068649"}]: dispatch 2024-09-17T00:18:26.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:26 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:18:26.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:26 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:18:26.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:26 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-09-17T00:18:26.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:26 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-09-17T00:18:26.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:26 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:18:26.968 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:26 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi002"}]: dispatch 2024-09-17T00:18:26.968 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:26 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:18:26.968 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:26 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:18:26.968 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:26 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:18:26.968 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:26 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:18:27.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:26 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:18:27.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:26 smithi002 ceph-mon[28203]: from='client.? ' entity='client.iscsi.foo.smithi002.vmouem' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6800/3807888742"}]': finished 2024-09-17T00:18:27.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:26 smithi002 ceph-mon[28203]: osdmap e83: 8 total, 8 up, 8 in 2024-09-17T00:18:27.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:26 smithi002 ceph-mon[28203]: from='client.? 172.21.15.2:0/312247546' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/1754068649"}]: dispatch 2024-09-17T00:18:27.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:26 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:18:27.230 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:26 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:18:27.230 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:26 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-09-17T00:18:27.230 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:26 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-09-17T00:18:27.230 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:26 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:18:27.230 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:26 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi002"}]: dispatch 2024-09-17T00:18:27.230 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:26 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:18:27.230 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:26 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:18:27.230 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:26 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:18:27.230 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:26 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:18:27.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:26 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:18:27.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:26 smithi002 ceph-mon[23737]: from='client.? ' entity='client.iscsi.foo.smithi002.vmouem' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6800/3807888742"}]': finished 2024-09-17T00:18:27.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:26 smithi002 ceph-mon[23737]: osdmap e83: 8 total, 8 up, 8 in 2024-09-17T00:18:27.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:26 smithi002 ceph-mon[23737]: from='client.? 172.21.15.2:0/312247546' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/1754068649"}]: dispatch 2024-09-17T00:18:27.231 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:26 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:18:27.231 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:26 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:18:27.231 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:26 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-09-17T00:18:27.231 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:26 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-09-17T00:18:27.231 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:26 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:18:27.231 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:26 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi002"}]: dispatch 2024-09-17T00:18:27.231 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:26 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:18:27.231 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:26 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:18:27.231 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:26 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:18:27.231 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:26 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:18:28.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:27 smithi099 ceph-mon[24594]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 9.3 KiB/s rd, 170 B/s wr, 12 op/s 2024-09-17T00:18:28.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:27 smithi099 ceph-mon[24594]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-09-17T00:18:28.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:27 smithi099 ceph-mon[24594]: Adding iSCSI gateway http://:@172.21.15.2:5000 to Dashboard 2024-09-17T00:18:28.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:27 smithi099 ceph-mon[24594]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-09-17T00:18:28.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:27 smithi099 ceph-mon[24594]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi002"}]: dispatch 2024-09-17T00:18:28.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:27 smithi099 ceph-mon[24594]: Checking dashboard <-> RGW credentials 2024-09-17T00:18:28.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:27 smithi099 ceph-mon[24594]: from='client.? 172.21.15.2:0/312247546' entity='client.iscsi.foo.smithi002.vmouem' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/1754068649"}]': finished 2024-09-17T00:18:28.218 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:27 smithi099 ceph-mon[24594]: osdmap e84: 8 total, 8 up, 8 in 2024-09-17T00:18:28.218 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:27 smithi099 ceph-mon[24594]: from='client.? 172.21.15.2:0/907480957' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/1780251331"}]: dispatch 2024-09-17T00:18:28.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:27 smithi002 ceph-mon[23737]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 9.3 KiB/s rd, 170 B/s wr, 12 op/s 2024-09-17T00:18:28.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:27 smithi002 ceph-mon[23737]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-09-17T00:18:28.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:27 smithi002 ceph-mon[23737]: Adding iSCSI gateway http://:@172.21.15.2:5000 to Dashboard 2024-09-17T00:18:28.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:27 smithi002 ceph-mon[23737]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-09-17T00:18:28.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:27 smithi002 ceph-mon[23737]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi002"}]: dispatch 2024-09-17T00:18:28.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:27 smithi002 ceph-mon[23737]: Checking dashboard <-> RGW credentials 2024-09-17T00:18:28.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:27 smithi002 ceph-mon[23737]: from='client.? 172.21.15.2:0/312247546' entity='client.iscsi.foo.smithi002.vmouem' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/1754068649"}]': finished 2024-09-17T00:18:28.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:27 smithi002 ceph-mon[23737]: osdmap e84: 8 total, 8 up, 8 in 2024-09-17T00:18:28.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:27 smithi002 ceph-mon[23737]: from='client.? 172.21.15.2:0/907480957' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/1780251331"}]: dispatch 2024-09-17T00:18:28.230 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:27 smithi002 ceph-mon[28203]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 9.3 KiB/s rd, 170 B/s wr, 12 op/s 2024-09-17T00:18:28.230 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:27 smithi002 ceph-mon[28203]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-09-17T00:18:28.230 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:27 smithi002 ceph-mon[28203]: Adding iSCSI gateway http://:@172.21.15.2:5000 to Dashboard 2024-09-17T00:18:28.230 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:27 smithi002 ceph-mon[28203]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-09-17T00:18:28.230 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:27 smithi002 ceph-mon[28203]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi002"}]: dispatch 2024-09-17T00:18:28.230 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:27 smithi002 ceph-mon[28203]: Checking dashboard <-> RGW credentials 2024-09-17T00:18:28.230 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:27 smithi002 ceph-mon[28203]: from='client.? 172.21.15.2:0/312247546' entity='client.iscsi.foo.smithi002.vmouem' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/1754068649"}]': finished 2024-09-17T00:18:28.230 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:27 smithi002 ceph-mon[28203]: osdmap e84: 8 total, 8 up, 8 in 2024-09-17T00:18:28.230 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:27 smithi002 ceph-mon[28203]: from='client.? 172.21.15.2:0/907480957' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/1780251331"}]: dispatch 2024-09-17T00:18:29.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:28 smithi099 ceph-mon[24594]: from='client.? 172.21.15.2:0/907480957' entity='client.iscsi.foo.smithi002.vmouem' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/1780251331"}]': finished 2024-09-17T00:18:29.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:28 smithi099 ceph-mon[24594]: osdmap e85: 8 total, 8 up, 8 in 2024-09-17T00:18:29.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:28 smithi099 ceph-mon[24594]: from='client.? 172.21.15.2:0/1292894876' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/1474587242"}]: dispatch 2024-09-17T00:18:29.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:28 smithi002 ceph-mon[23737]: from='client.? 172.21.15.2:0/907480957' entity='client.iscsi.foo.smithi002.vmouem' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/1780251331"}]': finished 2024-09-17T00:18:29.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:28 smithi002 ceph-mon[23737]: osdmap e85: 8 total, 8 up, 8 in 2024-09-17T00:18:29.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:28 smithi002 ceph-mon[23737]: from='client.? 172.21.15.2:0/1292894876' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/1474587242"}]: dispatch 2024-09-17T00:18:29.229 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:18:29 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: ::ffff:172.21.15.99 - - [17/Sep/2024:00:18:29] "GET /metrics HTTP/1.1" 200 215018 "" "Prometheus/2.33.4" 2024-09-17T00:18:29.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:28 smithi002 ceph-mon[28203]: from='client.? 172.21.15.2:0/907480957' entity='client.iscsi.foo.smithi002.vmouem' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/1780251331"}]': finished 2024-09-17T00:18:29.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:28 smithi002 ceph-mon[28203]: osdmap e85: 8 total, 8 up, 8 in 2024-09-17T00:18:29.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:28 smithi002 ceph-mon[28203]: from='client.? 172.21.15.2:0/1292894876' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/1474587242"}]: dispatch 2024-09-17T00:18:30.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:29 smithi099 ceph-mon[24594]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 255 B/s wr, 3 op/s 2024-09-17T00:18:30.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:29 smithi099 ceph-mon[24594]: from='client.? 172.21.15.2:0/1292894876' entity='client.iscsi.foo.smithi002.vmouem' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/1474587242"}]': finished 2024-09-17T00:18:30.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:29 smithi099 ceph-mon[24594]: osdmap e86: 8 total, 8 up, 8 in 2024-09-17T00:18:30.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:29 smithi099 ceph-mon[24594]: from='client.? 172.21.15.2:0/3559320287' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/824729147"}]: dispatch 2024-09-17T00:18:30.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:29 smithi099 ceph-mon[24594]: from='client.? ' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/824729147"}]: dispatch 2024-09-17T00:18:30.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:29 smithi002 ceph-mon[28203]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 255 B/s wr, 3 op/s 2024-09-17T00:18:30.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:29 smithi002 ceph-mon[28203]: from='client.? 172.21.15.2:0/1292894876' entity='client.iscsi.foo.smithi002.vmouem' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/1474587242"}]': finished 2024-09-17T00:18:30.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:29 smithi002 ceph-mon[28203]: osdmap e86: 8 total, 8 up, 8 in 2024-09-17T00:18:30.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:29 smithi002 ceph-mon[28203]: from='client.? 172.21.15.2:0/3559320287' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/824729147"}]: dispatch 2024-09-17T00:18:30.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:29 smithi002 ceph-mon[28203]: from='client.? ' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/824729147"}]: dispatch 2024-09-17T00:18:30.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:29 smithi002 ceph-mon[23737]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 255 B/s wr, 3 op/s 2024-09-17T00:18:30.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:29 smithi002 ceph-mon[23737]: from='client.? 172.21.15.2:0/1292894876' entity='client.iscsi.foo.smithi002.vmouem' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/1474587242"}]': finished 2024-09-17T00:18:30.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:29 smithi002 ceph-mon[23737]: osdmap e86: 8 total, 8 up, 8 in 2024-09-17T00:18:30.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:29 smithi002 ceph-mon[23737]: from='client.? 172.21.15.2:0/3559320287' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/824729147"}]: dispatch 2024-09-17T00:18:30.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:29 smithi002 ceph-mon[23737]: from='client.? ' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/824729147"}]: dispatch 2024-09-17T00:18:31.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:30 smithi099 ceph-mon[24594]: from='client.? ' entity='client.iscsi.foo.smithi002.vmouem' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/824729147"}]': finished 2024-09-17T00:18:31.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:30 smithi099 ceph-mon[24594]: osdmap e87: 8 total, 8 up, 8 in 2024-09-17T00:18:31.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:30 smithi099 ceph-mon[24594]: from='client.? 172.21.15.2:0/3505114452' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6801/2522360616"}]: dispatch 2024-09-17T00:18:31.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:30 smithi002 ceph-mon[28203]: from='client.? ' entity='client.iscsi.foo.smithi002.vmouem' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/824729147"}]': finished 2024-09-17T00:18:31.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:30 smithi002 ceph-mon[28203]: osdmap e87: 8 total, 8 up, 8 in 2024-09-17T00:18:31.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:30 smithi002 ceph-mon[28203]: from='client.? 172.21.15.2:0/3505114452' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6801/2522360616"}]: dispatch 2024-09-17T00:18:31.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:30 smithi002 ceph-mon[23737]: from='client.? ' entity='client.iscsi.foo.smithi002.vmouem' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/824729147"}]': finished 2024-09-17T00:18:31.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:30 smithi002 ceph-mon[23737]: osdmap e87: 8 total, 8 up, 8 in 2024-09-17T00:18:31.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:30 smithi002 ceph-mon[23737]: from='client.? 172.21.15.2:0/3505114452' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6801/2522360616"}]: dispatch 2024-09-17T00:18:32.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:31 smithi099 ceph-mon[24594]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:18:32.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:31 smithi099 ceph-mon[24594]: from='client.? 172.21.15.2:0/3505114452' entity='client.iscsi.foo.smithi002.vmouem' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6801/2522360616"}]': finished 2024-09-17T00:18:32.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:31 smithi099 ceph-mon[24594]: osdmap e88: 8 total, 8 up, 8 in 2024-09-17T00:18:32.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:31 smithi099 ceph-mon[24594]: from='client.? 172.21.15.2:0/162635742' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6800/2522360616"}]: dispatch 2024-09-17T00:18:32.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:31 smithi099 ceph-mon[24594]: from='client.? ' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6800/2522360616"}]: dispatch 2024-09-17T00:18:32.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:31 smithi002 ceph-mon[28203]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:18:32.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:31 smithi002 ceph-mon[28203]: from='client.? 172.21.15.2:0/3505114452' entity='client.iscsi.foo.smithi002.vmouem' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6801/2522360616"}]': finished 2024-09-17T00:18:32.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:31 smithi002 ceph-mon[28203]: osdmap e88: 8 total, 8 up, 8 in 2024-09-17T00:18:32.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:31 smithi002 ceph-mon[28203]: from='client.? 172.21.15.2:0/162635742' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6800/2522360616"}]: dispatch 2024-09-17T00:18:32.230 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:31 smithi002 ceph-mon[28203]: from='client.? ' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6800/2522360616"}]: dispatch 2024-09-17T00:18:32.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:31 smithi002 ceph-mon[23737]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:18:32.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:31 smithi002 ceph-mon[23737]: from='client.? 172.21.15.2:0/3505114452' entity='client.iscsi.foo.smithi002.vmouem' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6801/2522360616"}]': finished 2024-09-17T00:18:32.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:31 smithi002 ceph-mon[23737]: osdmap e88: 8 total, 8 up, 8 in 2024-09-17T00:18:32.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:31 smithi002 ceph-mon[23737]: from='client.? 172.21.15.2:0/162635742' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6800/2522360616"}]: dispatch 2024-09-17T00:18:32.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:31 smithi002 ceph-mon[23737]: from='client.? ' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6800/2522360616"}]: dispatch 2024-09-17T00:18:33.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:32 smithi099 ceph-mon[24594]: from='client.? ' entity='client.iscsi.foo.smithi002.vmouem' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6800/2522360616"}]': finished 2024-09-17T00:18:33.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:32 smithi099 ceph-mon[24594]: osdmap e89: 8 total, 8 up, 8 in 2024-09-17T00:18:33.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:32 smithi099 ceph-mon[24594]: from='client.? 172.21.15.2:0/837183868' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/3091500665"}]: dispatch 2024-09-17T00:18:33.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:32 smithi002 ceph-mon[28203]: from='client.? ' entity='client.iscsi.foo.smithi002.vmouem' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6800/2522360616"}]': finished 2024-09-17T00:18:33.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:32 smithi002 ceph-mon[28203]: osdmap e89: 8 total, 8 up, 8 in 2024-09-17T00:18:33.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:32 smithi002 ceph-mon[28203]: from='client.? 172.21.15.2:0/837183868' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/3091500665"}]: dispatch 2024-09-17T00:18:33.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:32 smithi002 ceph-mon[23737]: from='client.? ' entity='client.iscsi.foo.smithi002.vmouem' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6800/2522360616"}]': finished 2024-09-17T00:18:33.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:32 smithi002 ceph-mon[23737]: osdmap e89: 8 total, 8 up, 8 in 2024-09-17T00:18:33.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:32 smithi002 ceph-mon[23737]: from='client.? 172.21.15.2:0/837183868' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/3091500665"}]: dispatch 2024-09-17T00:18:33.863 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Sep 17 00:18:33 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-alertmanager-a[48044]: level=error ts=2024-09-17T00:18:33.509Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.99:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.99 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.2:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.2 because it doesn't contain any IP SANs" 2024-09-17T00:18:33.863 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Sep 17 00:18:33 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-alertmanager-a[48044]: level=warn ts=2024-09-17T00:18:33.511Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.2:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.2 because it doesn't contain any IP SANs" 2024-09-17T00:18:33.864 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Sep 17 00:18:33 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-alertmanager-a[48044]: level=warn ts=2024-09-17T00:18:33.511Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.99:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.99 because it doesn't contain any IP SANs" 2024-09-17T00:18:34.216 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:33 smithi099 ceph-mon[24594]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 9 op/s 2024-09-17T00:18:34.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:33 smithi099 ceph-mon[24594]: from='client.14889 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:18:34.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:33 smithi099 ceph-mon[24594]: from='client.? 172.21.15.2:0/837183868' entity='client.iscsi.foo.smithi002.vmouem' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/3091500665"}]': finished 2024-09-17T00:18:34.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:33 smithi099 ceph-mon[24594]: osdmap e90: 8 total, 8 up, 8 in 2024-09-17T00:18:34.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:33 smithi099 ceph-mon[24594]: from='client.? 172.21.15.2:0/3935740507' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/3536227112"}]: dispatch 2024-09-17T00:18:34.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:33 smithi002 ceph-mon[23737]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 9 op/s 2024-09-17T00:18:34.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:33 smithi002 ceph-mon[23737]: from='client.14889 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:18:34.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:33 smithi002 ceph-mon[23737]: from='client.? 172.21.15.2:0/837183868' entity='client.iscsi.foo.smithi002.vmouem' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/3091500665"}]': finished 2024-09-17T00:18:34.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:33 smithi002 ceph-mon[23737]: osdmap e90: 8 total, 8 up, 8 in 2024-09-17T00:18:34.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:33 smithi002 ceph-mon[23737]: from='client.? 172.21.15.2:0/3935740507' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/3536227112"}]: dispatch 2024-09-17T00:18:34.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:33 smithi002 ceph-mon[28203]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 9 op/s 2024-09-17T00:18:34.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:33 smithi002 ceph-mon[28203]: from='client.14889 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:18:34.230 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:33 smithi002 ceph-mon[28203]: from='client.? 172.21.15.2:0/837183868' entity='client.iscsi.foo.smithi002.vmouem' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/3091500665"}]': finished 2024-09-17T00:18:34.230 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:33 smithi002 ceph-mon[28203]: osdmap e90: 8 total, 8 up, 8 in 2024-09-17T00:18:34.230 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:33 smithi002 ceph-mon[28203]: from='client.? 172.21.15.2:0/3935740507' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/3536227112"}]: dispatch 2024-09-17T00:18:35.130 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:34 smithi002 ceph-mon[28203]: from='client.? 172.21.15.2:0/3935740507' entity='client.iscsi.foo.smithi002.vmouem' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/3536227112"}]': finished 2024-09-17T00:18:35.130 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:34 smithi002 ceph-mon[28203]: osdmap e91: 8 total, 8 up, 8 in 2024-09-17T00:18:35.131 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:34 smithi002 ceph-mon[28203]: from='client.? 172.21.15.2:0/161124815' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6800/3667297605"}]: dispatch 2024-09-17T00:18:35.131 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:34 smithi002 ceph-mon[28203]: from='client.? ' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6800/3667297605"}]: dispatch 2024-09-17T00:18:35.131 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:34 smithi002 ceph-mon[23737]: from='client.? 172.21.15.2:0/3935740507' entity='client.iscsi.foo.smithi002.vmouem' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/3536227112"}]': finished 2024-09-17T00:18:35.131 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:34 smithi002 ceph-mon[23737]: osdmap e91: 8 total, 8 up, 8 in 2024-09-17T00:18:35.131 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:34 smithi002 ceph-mon[23737]: from='client.? 172.21.15.2:0/161124815' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6800/3667297605"}]: dispatch 2024-09-17T00:18:35.131 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:34 smithi002 ceph-mon[23737]: from='client.? ' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6800/3667297605"}]: dispatch 2024-09-17T00:18:35.216 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:18:34 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[25772]: ::ffff:172.21.15.99 - - [17/Sep/2024:00:18:34] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-09-17T00:18:35.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:34 smithi099 ceph-mon[24594]: from='client.? 172.21.15.2:0/3935740507' entity='client.iscsi.foo.smithi002.vmouem' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/3536227112"}]': finished 2024-09-17T00:18:35.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:34 smithi099 ceph-mon[24594]: osdmap e91: 8 total, 8 up, 8 in 2024-09-17T00:18:35.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:34 smithi099 ceph-mon[24594]: from='client.? 172.21.15.2:0/161124815' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6800/3667297605"}]: dispatch 2024-09-17T00:18:35.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:34 smithi099 ceph-mon[24594]: from='client.? ' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6800/3667297605"}]: dispatch 2024-09-17T00:18:36.142 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:35 smithi002 ceph-mon[28203]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 9 op/s 2024-09-17T00:18:36.142 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:35 smithi002 ceph-mon[28203]: from='client.? ' entity='client.iscsi.foo.smithi002.vmouem' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6800/3667297605"}]': finished 2024-09-17T00:18:36.143 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:35 smithi002 ceph-mon[28203]: osdmap e92: 8 total, 8 up, 8 in 2024-09-17T00:18:36.143 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:35 smithi002 ceph-mon[28203]: from='client.? 172.21.15.2:0/1972130515' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/626695359"}]: dispatch 2024-09-17T00:18:36.143 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:35 smithi002 ceph-mon[23737]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 9 op/s 2024-09-17T00:18:36.143 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:35 smithi002 ceph-mon[23737]: from='client.? ' entity='client.iscsi.foo.smithi002.vmouem' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6800/3667297605"}]': finished 2024-09-17T00:18:36.143 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:35 smithi002 ceph-mon[23737]: osdmap e92: 8 total, 8 up, 8 in 2024-09-17T00:18:36.143 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:35 smithi002 ceph-mon[23737]: from='client.? 172.21.15.2:0/1972130515' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/626695359"}]: dispatch 2024-09-17T00:18:36.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:35 smithi099 ceph-mon[24594]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 9 op/s 2024-09-17T00:18:36.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:35 smithi099 ceph-mon[24594]: from='client.? ' entity='client.iscsi.foo.smithi002.vmouem' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6800/3667297605"}]': finished 2024-09-17T00:18:36.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:35 smithi099 ceph-mon[24594]: osdmap e92: 8 total, 8 up, 8 in 2024-09-17T00:18:36.218 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:35 smithi099 ceph-mon[24594]: from='client.? 172.21.15.2:0/1972130515' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/626695359"}]: dispatch 2024-09-17T00:18:37.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:36 smithi099 ceph-mon[24594]: from='client.? 172.21.15.2:0/1972130515' entity='client.iscsi.foo.smithi002.vmouem' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/626695359"}]': finished 2024-09-17T00:18:37.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:36 smithi099 ceph-mon[24594]: osdmap e93: 8 total, 8 up, 8 in 2024-09-17T00:18:37.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:36 smithi099 ceph-mon[24594]: from='client.? 172.21.15.2:0/602764985' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6801/3667297605"}]: dispatch 2024-09-17T00:18:37.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:36 smithi002 ceph-mon[23737]: from='client.? 172.21.15.2:0/1972130515' entity='client.iscsi.foo.smithi002.vmouem' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/626695359"}]': finished 2024-09-17T00:18:37.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:36 smithi002 ceph-mon[23737]: osdmap e93: 8 total, 8 up, 8 in 2024-09-17T00:18:37.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:36 smithi002 ceph-mon[23737]: from='client.? 172.21.15.2:0/602764985' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6801/3667297605"}]: dispatch 2024-09-17T00:18:37.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:36 smithi002 ceph-mon[28203]: from='client.? 172.21.15.2:0/1972130515' entity='client.iscsi.foo.smithi002.vmouem' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/626695359"}]': finished 2024-09-17T00:18:37.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:36 smithi002 ceph-mon[28203]: osdmap e93: 8 total, 8 up, 8 in 2024-09-17T00:18:37.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:36 smithi002 ceph-mon[28203]: from='client.? 172.21.15.2:0/602764985' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6801/3667297605"}]: dispatch 2024-09-17T00:18:38.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:37 smithi099 ceph-mon[24594]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:18:38.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:37 smithi099 ceph-mon[24594]: from='client.? 172.21.15.2:0/602764985' entity='client.iscsi.foo.smithi002.vmouem' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6801/3667297605"}]': finished 2024-09-17T00:18:38.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:37 smithi099 ceph-mon[24594]: osdmap e94: 8 total, 8 up, 8 in 2024-09-17T00:18:38.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:37 smithi099 ceph-mon[24594]: from='client.? 172.21.15.2:0/2388044738' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/3867827864"}]: dispatch 2024-09-17T00:18:38.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:37 smithi002 ceph-mon[23737]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:18:38.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:37 smithi002 ceph-mon[23737]: from='client.? 172.21.15.2:0/602764985' entity='client.iscsi.foo.smithi002.vmouem' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6801/3667297605"}]': finished 2024-09-17T00:18:38.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:37 smithi002 ceph-mon[23737]: osdmap e94: 8 total, 8 up, 8 in 2024-09-17T00:18:38.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:37 smithi002 ceph-mon[23737]: from='client.? 172.21.15.2:0/2388044738' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/3867827864"}]: dispatch 2024-09-17T00:18:38.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:37 smithi002 ceph-mon[28203]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:18:38.230 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:37 smithi002 ceph-mon[28203]: from='client.? 172.21.15.2:0/602764985' entity='client.iscsi.foo.smithi002.vmouem' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6801/3667297605"}]': finished 2024-09-17T00:18:38.230 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:37 smithi002 ceph-mon[28203]: osdmap e94: 8 total, 8 up, 8 in 2024-09-17T00:18:38.230 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:37 smithi002 ceph-mon[28203]: from='client.? 172.21.15.2:0/2388044738' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/3867827864"}]: dispatch 2024-09-17T00:18:39.216 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:38 smithi099 ceph-mon[24594]: from='client.? 172.21.15.2:0/2388044738' entity='client.iscsi.foo.smithi002.vmouem' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/3867827864"}]': finished 2024-09-17T00:18:39.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:38 smithi099 ceph-mon[24594]: osdmap e95: 8 total, 8 up, 8 in 2024-09-17T00:18:39.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:38 smithi099 ceph-mon[24594]: from='client.? 172.21.15.2:0/1825192993' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/174032459"}]: dispatch 2024-09-17T00:18:39.229 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:18:39 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: ::ffff:172.21.15.99 - - [17/Sep/2024:00:18:39] "GET /metrics HTTP/1.1" 200 215018 "" "Prometheus/2.33.4" 2024-09-17T00:18:39.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:38 smithi002 ceph-mon[23737]: from='client.? 172.21.15.2:0/2388044738' entity='client.iscsi.foo.smithi002.vmouem' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/3867827864"}]': finished 2024-09-17T00:18:39.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:38 smithi002 ceph-mon[23737]: osdmap e95: 8 total, 8 up, 8 in 2024-09-17T00:18:39.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:38 smithi002 ceph-mon[23737]: from='client.? 172.21.15.2:0/1825192993' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/174032459"}]: dispatch 2024-09-17T00:18:39.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:38 smithi002 ceph-mon[28203]: from='client.? 172.21.15.2:0/2388044738' entity='client.iscsi.foo.smithi002.vmouem' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/3867827864"}]': finished 2024-09-17T00:18:39.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:38 smithi002 ceph-mon[28203]: osdmap e95: 8 total, 8 up, 8 in 2024-09-17T00:18:39.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:38 smithi002 ceph-mon[28203]: from='client.? 172.21.15.2:0/1825192993' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/174032459"}]: dispatch 2024-09-17T00:18:40.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:39 smithi099 ceph-mon[24594]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:18:40.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:39 smithi099 ceph-mon[24594]: from='client.? 172.21.15.2:0/1825192993' entity='client.iscsi.foo.smithi002.vmouem' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/174032459"}]': finished 2024-09-17T00:18:40.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:39 smithi099 ceph-mon[24594]: osdmap e96: 8 total, 8 up, 8 in 2024-09-17T00:18:40.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:39 smithi002 ceph-mon[23737]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:18:40.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:39 smithi002 ceph-mon[23737]: from='client.? 172.21.15.2:0/1825192993' entity='client.iscsi.foo.smithi002.vmouem' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/174032459"}]': finished 2024-09-17T00:18:40.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:39 smithi002 ceph-mon[23737]: osdmap e96: 8 total, 8 up, 8 in 2024-09-17T00:18:40.230 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:39 smithi002 ceph-mon[28203]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:18:40.230 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:39 smithi002 ceph-mon[28203]: from='client.? 172.21.15.2:0/1825192993' entity='client.iscsi.foo.smithi002.vmouem' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/174032459"}]': finished 2024-09-17T00:18:40.230 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:39 smithi002 ceph-mon[28203]: osdmap e96: 8 total, 8 up, 8 in 2024-09-17T00:18:42.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:41 smithi099 ceph-mon[24594]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-17T00:18:42.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:41 smithi002 ceph-mon[23737]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-17T00:18:42.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:41 smithi002 ceph-mon[28203]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-17T00:18:43.905 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Sep 17 00:18:43 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-alertmanager-a[48044]: level=error ts=2024-09-17T00:18:43.510Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.99:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.99 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.2:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.2 because it doesn't contain any IP SANs" 2024-09-17T00:18:43.905 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Sep 17 00:18:43 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-alertmanager-a[48044]: level=warn ts=2024-09-17T00:18:43.512Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.2:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.2 because it doesn't contain any IP SANs" 2024-09-17T00:18:43.905 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Sep 17 00:18:43 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-alertmanager-a[48044]: level=warn ts=2024-09-17T00:18:43.512Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.99:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.99 because it doesn't contain any IP SANs" 2024-09-17T00:18:44.216 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:43 smithi099 ceph-mon[24594]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:18:44.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:43 smithi099 ceph-mon[24594]: from='client.14889 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:18:44.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:43 smithi002 ceph-mon[23737]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:18:44.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:43 smithi002 ceph-mon[23737]: from='client.14889 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:18:44.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:43 smithi002 ceph-mon[28203]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:18:44.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:43 smithi002 ceph-mon[28203]: from='client.14889 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:18:45.217 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:18:44 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[25772]: ::ffff:172.21.15.99 - - [17/Sep/2024:00:18:44] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-09-17T00:18:46.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:45 smithi099 ceph-mon[24594]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 683 B/s rd, 0 op/s 2024-09-17T00:18:46.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:45 smithi002 ceph-mon[23737]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 683 B/s rd, 0 op/s 2024-09-17T00:18:46.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:45 smithi002 ceph-mon[28203]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 683 B/s rd, 0 op/s 2024-09-17T00:18:48.216 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:47 smithi099 ceph-mon[24594]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:18:48.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:47 smithi002 ceph-mon[23737]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:18:48.230 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:47 smithi002 ceph-mon[28203]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:18:49.479 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:18:49 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: ::ffff:172.21.15.99 - - [17/Sep/2024:00:18:49] "GET /metrics HTTP/1.1" 200 214996 "" "Prometheus/2.33.4" 2024-09-17T00:18:50.216 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:49 smithi099 ceph-mon[24594]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-17T00:18:50.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:49 smithi002 ceph-mon[28203]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-17T00:18:50.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:49 smithi002 ceph-mon[23737]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-17T00:18:52.216 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:51 smithi099 ceph-mon[24594]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 892 B/s rd, 0 op/s 2024-09-17T00:18:52.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:51 smithi002 ceph-mon[23737]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 892 B/s rd, 0 op/s 2024-09-17T00:18:52.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:51 smithi002 ceph-mon[28203]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 892 B/s rd, 0 op/s 2024-09-17T00:18:53.919 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Sep 17 00:18:53 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-alertmanager-a[48044]: level=error ts=2024-09-17T00:18:53.511Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.99:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.99 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.2:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.2 because it doesn't contain any IP SANs" 2024-09-17T00:18:54.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:53 smithi099 ceph-mon[24594]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:18:54.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:53 smithi099 ceph-mon[24594]: from='client.14889 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:18:54.228 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:53 smithi002 ceph-mon[23737]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:18:54.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:53 smithi002 ceph-mon[23737]: from='client.14889 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:18:54.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:53 smithi002 ceph-mon[28203]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:18:54.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:53 smithi002 ceph-mon[28203]: from='client.14889 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:18:55.216 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:18:54 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[25772]: ::ffff:172.21.15.99 - - [17/Sep/2024:00:18:54] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-09-17T00:18:56.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:55 smithi099 ceph-mon[24594]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:18:56.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:55 smithi002 ceph-mon[28203]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:18:56.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:55 smithi002 ceph-mon[23737]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:18:58.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:57 smithi099 ceph-mon[24594]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:18:58.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:57 smithi002 ceph-mon[28203]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:18:58.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:57 smithi002 ceph-mon[23737]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:18:59.479 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:18:59 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: ::ffff:172.21.15.99 - - [17/Sep/2024:00:18:59] "GET /metrics HTTP/1.1" 200 214946 "" "Prometheus/2.33.4" 2024-09-17T00:19:00.216 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:18:59 smithi099 ceph-mon[24594]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:19:00.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:18:59 smithi002 ceph-mon[23737]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:19:00.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:18:59 smithi002 ceph-mon[28203]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:19:02.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:19:01 smithi099 ceph-mon[24594]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:19:02.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:19:01 smithi002 ceph-mon[28203]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:19:02.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:19:01 smithi002 ceph-mon[23737]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:19:04.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:19:03 smithi099 ceph-mon[24594]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:19:04.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:19:03 smithi099 ceph-mon[24594]: from='client.14889 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:19:04.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:19:03 smithi002 ceph-mon[28203]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:19:04.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:19:03 smithi002 ceph-mon[28203]: from='client.14889 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:19:04.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:19:03 smithi002 ceph-mon[23737]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:19:04.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:19:03 smithi002 ceph-mon[23737]: from='client.14889 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:19:05.216 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:19:04 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[25772]: ::ffff:172.21.15.99 - - [17/Sep/2024:00:19:04] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-09-17T00:19:06.216 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:19:05 smithi099 ceph-mon[24594]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:19:06.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:19:05 smithi002 ceph-mon[28203]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:19:06.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:19:05 smithi002 ceph-mon[23737]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:19:08.216 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:19:07 smithi099 ceph-mon[24594]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:19:08.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:19:07 smithi002 ceph-mon[28203]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:19:08.293 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:19:07 smithi002 ceph-mon[23737]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:19:09.479 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:19:09 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: ::ffff:172.21.15.99 - - [17/Sep/2024:00:19:09] "GET /metrics HTTP/1.1" 200 214946 "" "Prometheus/2.33.4" 2024-09-17T00:19:10.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:19:09 smithi099 ceph-mon[24594]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:19:10.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:19:09 smithi002 ceph-mon[23737]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:19:10.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:19:09 smithi002 ceph-mon[28203]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:19:11.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:19:10 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T00:19:11.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:19:10 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T00:19:11.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:19:10 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T00:19:11.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:19:10 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T00:19:11.230 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:19:10 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T00:19:11.230 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:19:10 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T00:19:12.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:19:11 smithi099 ceph-mon[24594]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:19:12.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:19:11 smithi002 ceph-mon[28203]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:19:12.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:19:11 smithi002 ceph-mon[23737]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:19:14.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:19:13 smithi099 ceph-mon[24594]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:19:14.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:19:13 smithi099 ceph-mon[24594]: from='client.14889 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:19:14.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:19:13 smithi002 ceph-mon[23737]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:19:14.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:19:13 smithi002 ceph-mon[23737]: from='client.14889 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:19:14.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:19:13 smithi002 ceph-mon[28203]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:19:14.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:19:13 smithi002 ceph-mon[28203]: from='client.14889 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:19:15.217 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:19:14 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[25772]: ::ffff:172.21.15.99 - - [17/Sep/2024:00:19:14] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-09-17T00:19:16.216 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:19:15 smithi099 ceph-mon[24594]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:19:16.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:19:15 smithi002 ceph-mon[28203]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:19:16.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:19:15 smithi002 ceph-mon[23737]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:19:18.216 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:19:17 smithi099 ceph-mon[24594]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:19:18.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:19:17 smithi002 ceph-mon[23737]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:19:18.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:19:17 smithi002 ceph-mon[28203]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:19:19.479 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:19:19 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: ::ffff:172.21.15.99 - - [17/Sep/2024:00:19:19] "GET /metrics HTTP/1.1" 200 214922 "" "Prometheus/2.33.4" 2024-09-17T00:19:20.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:19:19 smithi099 ceph-mon[24594]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:19:20.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:19:19 smithi002 ceph-mon[23737]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:19:20.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:19:19 smithi002 ceph-mon[28203]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:19:22.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:19:21 smithi099 ceph-mon[24594]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:19:22.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:19:21 smithi002 ceph-mon[28203]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:19:22.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:19:21 smithi002 ceph-mon[23737]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:19:24.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:19:23 smithi002 ceph-mon[23737]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:19:24.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:19:23 smithi002 ceph-mon[23737]: from='client.14889 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:19:24.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:19:23 smithi002 ceph-mon[28203]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:19:24.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:19:23 smithi002 ceph-mon[28203]: from='client.14889 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:19:24.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:19:23 smithi099 ceph-mon[24594]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:19:24.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:19:23 smithi099 ceph-mon[24594]: from='client.14889 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:19:25.217 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:19:24 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[25772]: ::ffff:172.21.15.99 - - [17/Sep/2024:00:19:24] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-09-17T00:19:26.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:19:25 smithi002 ceph-mon[23737]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:19:26.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:19:25 smithi002 ceph-mon[28203]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:19:26.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:19:25 smithi099 ceph-mon[24594]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:19:27.010 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:19:26 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:19:27.010 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:19:26 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:19:27.028 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:19:26 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:19:27.028 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:19:26 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:19:27.028 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:19:26 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:19:27.028 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:19:26 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:19:28.466 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:19:28 smithi099 ceph-mon[24594]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:19:28.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:19:28 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:19:28.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:19:28 smithi002 ceph-mon[23737]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:19:28.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:19:28 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:19:28.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:19:28 smithi002 ceph-mon[28203]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:19:28.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:19:28 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:19:29.479 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:19:29 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: ::ffff:172.21.15.99 - - [17/Sep/2024:00:19:29] "GET /metrics HTTP/1.1" 200 214911 "" "Prometheus/2.33.4" 2024-09-17T00:19:30.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:19:30 smithi099 ceph-mon[24594]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:19:30.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:19:30 smithi002 ceph-mon[23737]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:19:30.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:19:30 smithi002 ceph-mon[28203]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:19:32.466 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:19:32 smithi099 ceph-mon[24594]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:19:32.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:19:32 smithi002 ceph-mon[28203]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:19:32.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:19:32 smithi002 ceph-mon[23737]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:19:34.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:19:34 smithi099 ceph-mon[24594]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:19:34.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:19:34 smithi099 ceph-mon[24594]: from='client.14889 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:19:34.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:19:34 smithi002 ceph-mon[23737]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:19:34.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:19:34 smithi002 ceph-mon[23737]: from='client.14889 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:19:34.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:19:34 smithi002 ceph-mon[28203]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:19:34.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:19:34 smithi002 ceph-mon[28203]: from='client.14889 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:19:35.216 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:19:34 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[25772]: ::ffff:172.21.15.99 - - [17/Sep/2024:00:19:34] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-09-17T00:19:36.466 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:19:36 smithi099 ceph-mon[24594]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:19:36.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:19:36 smithi002 ceph-mon[28203]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:19:36.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:19:36 smithi002 ceph-mon[23737]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:19:38.466 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:19:38 smithi099 ceph-mon[24594]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:19:38.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:19:38 smithi002 ceph-mon[28203]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:19:38.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:19:38 smithi002 ceph-mon[23737]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:19:39.479 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:19:39 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: ::ffff:172.21.15.99 - - [17/Sep/2024:00:19:39] "GET /metrics HTTP/1.1" 200 214911 "" "Prometheus/2.33.4" 2024-09-17T00:19:40.466 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:19:40 smithi099 ceph-mon[24594]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:19:40.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:19:40 smithi002 ceph-mon[28203]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:19:40.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:19:40 smithi002 ceph-mon[23737]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:19:42.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:19:42 smithi099 ceph-mon[24594]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:19:42.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:19:42 smithi002 ceph-mon[28203]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:19:42.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:19:42 smithi002 ceph-mon[23737]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:19:44.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:19:44 smithi099 ceph-mon[24594]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:19:44.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:19:44 smithi099 ceph-mon[24594]: from='client.14889 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:19:44.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:19:44 smithi002 ceph-mon[23737]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:19:44.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:19:44 smithi002 ceph-mon[23737]: from='client.14889 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:19:44.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:19:44 smithi002 ceph-mon[28203]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:19:44.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:19:44 smithi002 ceph-mon[28203]: from='client.14889 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:19:45.217 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:19:44 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[25772]: ::ffff:172.21.15.99 - - [17/Sep/2024:00:19:44] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-09-17T00:19:46.466 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:19:46 smithi099 ceph-mon[24594]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:19:46.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:19:46 smithi002 ceph-mon[23737]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:19:46.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:19:46 smithi002 ceph-mon[28203]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:19:48.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:19:48 smithi099 ceph-mon[24594]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:19:48.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:19:48 smithi002 ceph-mon[23737]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:19:48.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:19:48 smithi002 ceph-mon[28203]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:19:49.479 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:19:49 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: ::ffff:172.21.15.99 - - [17/Sep/2024:00:19:49] "GET /metrics HTTP/1.1" 200 214912 "" "Prometheus/2.33.4" 2024-09-17T00:19:50.466 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:19:50 smithi099 ceph-mon[24594]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:19:50.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:19:50 smithi002 ceph-mon[28203]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:19:50.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:19:50 smithi002 ceph-mon[23737]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:19:52.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:19:52 smithi099 ceph-mon[24594]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:19:52.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:19:52 smithi002 ceph-mon[23737]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:19:52.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:19:52 smithi002 ceph-mon[28203]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:19:54.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:19:54 smithi099 ceph-mon[24594]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:19:54.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:19:54 smithi099 ceph-mon[24594]: from='client.14889 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:19:54.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:19:54 smithi002 ceph-mon[23737]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:19:54.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:19:54 smithi002 ceph-mon[23737]: from='client.14889 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:19:54.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:19:54 smithi002 ceph-mon[28203]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:19:54.480 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:19:54 smithi002 ceph-mon[28203]: from='client.14889 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:19:55.217 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:19:54 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[25772]: ::ffff:172.21.15.99 - - [17/Sep/2024:00:19:54] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-09-17T00:19:56.389 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:19:56 smithi099 ceph-mon[24594]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:19:56.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:19:56 smithi002 ceph-mon[23737]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:19:56.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:19:56 smithi002 ceph-mon[28203]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:19:58.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:19:58 smithi099 ceph-mon[24594]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:19:58.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:19:58 smithi002 ceph-mon[23737]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:19:58.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:19:58 smithi002 ceph-mon[28203]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:19:59.479 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:19:59 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: ::ffff:172.21.15.99 - - [17/Sep/2024:00:19:59] "GET /metrics HTTP/1.1" 200 214917 "" "Prometheus/2.33.4" 2024-09-17T00:20:00.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:20:00 smithi099 ceph-mon[24594]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:20:00.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:20:00 smithi099 ceph-mon[24594]: overall HEALTH_OK 2024-09-17T00:20:00.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:20:00 smithi002 ceph-mon[28203]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:20:00.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:20:00 smithi002 ceph-mon[28203]: overall HEALTH_OK 2024-09-17T00:20:00.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:20:00 smithi002 ceph-mon[23737]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:20:00.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:20:00 smithi002 ceph-mon[23737]: overall HEALTH_OK 2024-09-17T00:20:02.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:20:02 smithi099 ceph-mon[24594]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:20:02.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:20:02 smithi002 ceph-mon[23737]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:20:02.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:20:02 smithi002 ceph-mon[28203]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:20:04.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:20:04 smithi099 ceph-mon[24594]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:20:04.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:20:04 smithi099 ceph-mon[24594]: from='client.14889 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:20:04.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:20:04 smithi002 ceph-mon[23737]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:20:04.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:20:04 smithi002 ceph-mon[23737]: from='client.14889 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:20:04.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:20:04 smithi002 ceph-mon[28203]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:20:04.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:20:04 smithi002 ceph-mon[28203]: from='client.14889 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:20:05.217 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:20:04 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[25772]: ::ffff:172.21.15.99 - - [17/Sep/2024:00:20:04] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-09-17T00:20:06.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:20:06 smithi099 ceph-mon[24594]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:20:06.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:20:06 smithi002 ceph-mon[28203]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:20:06.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:20:06 smithi002 ceph-mon[23737]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:20:08.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:20:08 smithi099 ceph-mon[24594]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:20:08.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:20:08 smithi002 ceph-mon[28203]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:20:08.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:20:08 smithi002 ceph-mon[23737]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:20:09.479 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:20:09 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: ::ffff:172.21.15.99 - - [17/Sep/2024:00:20:09] "GET /metrics HTTP/1.1" 200 214917 "" "Prometheus/2.33.4" 2024-09-17T00:20:10.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:20:10 smithi099 ceph-mon[24594]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:20:10.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:20:10 smithi002 ceph-mon[23737]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:20:10.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:20:10 smithi002 ceph-mon[28203]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:20:11.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:20:11 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T00:20:11.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:20:11 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T00:20:11.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:20:11 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T00:20:11.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:20:11 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T00:20:11.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:20:11 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T00:20:11.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:20:11 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T00:20:12.466 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:20:12 smithi099 ceph-mon[24594]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:20:12.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:20:12 smithi002 ceph-mon[23737]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:20:12.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:20:12 smithi002 ceph-mon[28203]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:20:14.466 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:20:14 smithi099 ceph-mon[24594]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:20:14.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:20:14 smithi099 ceph-mon[24594]: from='client.14889 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:20:14.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:20:14 smithi002 ceph-mon[23737]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:20:14.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:20:14 smithi002 ceph-mon[23737]: from='client.14889 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:20:14.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:20:14 smithi002 ceph-mon[28203]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:20:14.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:20:14 smithi002 ceph-mon[28203]: from='client.14889 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:20:15.217 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:20:14 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[25772]: ::ffff:172.21.15.99 - - [17/Sep/2024:00:20:14] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-09-17T00:20:16.144 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3aad27a2-7489-11ef-bceb-c7b262605968 -e sha1=c2ddf9721ef254645201cdbabd9c8f49c30a586f -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force' 2024-09-17T00:20:16.458 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:20:16 smithi002 ceph-mon[28203]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:20:16.459 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:20:16 smithi002 ceph-mon[23737]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:20:16.466 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:20:16 smithi099 ceph-mon[24594]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:20:18.448 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:20:18 smithi002 ceph-mon[23737]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:20:18.448 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:20:18 smithi002 ceph-mon[28203]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:20:18.466 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:20:18 smithi099 ceph-mon[24594]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:20:18.797 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3aad27a2-7489-11ef-bceb-c7b262605968 -e sha1=c2ddf9721ef254645201cdbabd9c8f49c30a586f -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force' 2024-09-17T00:20:19.397 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:20:19 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: ::ffff:172.21.15.99 - - [17/Sep/2024:00:20:19] "GET /metrics HTTP/1.1" 200 214932 "" "Prometheus/2.33.4" 2024-09-17T00:20:20.385 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:20:20 smithi002 ceph-mon[28203]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:20:20.386 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:20:20 smithi002 ceph-mon[23737]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:20:20.466 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:20:20 smithi099 ceph-mon[24594]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:20:21.470 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3aad27a2-7489-11ef-bceb-c7b262605968 -e sha1=c2ddf9721ef254645201cdbabd9c8f49c30a586f -- bash -c 'ceph config set global log_to_journald false --force' 2024-09-17T00:20:22.466 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:20:22 smithi002 ceph-mon[28203]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:20:22.466 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:20:22 smithi002 ceph-mon[23737]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:20:22.466 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:20:22 smithi099 ceph-mon[24594]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:20:24.064 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3aad27a2-7489-11ef-bceb-c7b262605968 -e sha1=c2ddf9721ef254645201cdbabd9c8f49c30a586f -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1' 2024-09-17T00:20:24.370 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:20:24 smithi002 ceph-mon[28203]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:20:24.370 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:20:24 smithi002 ceph-mon[28203]: from='client.14889 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:20:24.370 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:20:24 smithi002 ceph-mon[23737]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:20:24.370 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:20:24 smithi002 ceph-mon[23737]: from='client.14889 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:20:24.466 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:20:24 smithi099 ceph-mon[24594]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:20:24.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:20:24 smithi099 ceph-mon[24594]: from='client.14889 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:20:25.217 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:20:24 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[25772]: ::ffff:172.21.15.99 - - [17/Sep/2024:00:20:24] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-09-17T00:20:26.016 INFO:teuthology.orchestra.run.smithi002.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f 2024-09-17T00:20:26.450 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:20:26 smithi002 ceph-mon[28203]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:20:26.450 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:20:26 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:20:26.450 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:20:26 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:20:26.450 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:20:26 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:20:26.450 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:20:26 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:20:26.450 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:20:26 smithi002 ceph-mon[23737]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:20:26.451 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:20:26 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:20:26.451 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:20:26 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:20:26.451 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:20:26 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:20:26.451 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:20:26 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:20:26.466 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:20:26 smithi099 ceph-mon[24594]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:20:26.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:20:26 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:20:26.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:20:26 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:20:26.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:20:26 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:20:26.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:20:26 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:20:26.669 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-09-17T00:20:26.679 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi002.front.sepia.ceph.com 2024-09-17T00:20:26.679 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3aad27a2-7489-11ef-bceb-c7b262605968 -e sha1=c2ddf9721ef254645201cdbabd9c8f49c30a586f -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done' 2024-09-17T00:20:27.309 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:20:27 smithi002 ceph-mon[23737]: from='client.24934 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:20:27.309 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:20:27 smithi002 ceph-mon[23737]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f 2024-09-17T00:20:27.309 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:20:27 smithi002 ceph-mon[23737]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f 2024-09-17T00:20:27.309 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:20:27 smithi002 ceph-mon[28203]: from='client.24934 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:20:27.309 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:20:27 smithi002 ceph-mon[28203]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f 2024-09-17T00:20:27.310 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:20:27 smithi002 ceph-mon[28203]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f 2024-09-17T00:20:27.466 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:20:27 smithi099 ceph-mon[24594]: from='client.24934 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:20:27.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:20:27 smithi099 ceph-mon[24594]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f 2024-09-17T00:20:27.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:20:27 smithi099 ceph-mon[24594]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f 2024-09-17T00:20:28.389 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:20:28 smithi002 ceph-mon[28203]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:20:28.389 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:20:28 smithi002 ceph-mon[23737]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:20:28.466 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:20:28 smithi099 ceph-mon[24594]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:20:28.934 INFO:teuthology.orchestra.run.smithi002.stdout:true 2024-09-17T00:20:29.229 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:20:29 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: ::ffff:172.21.15.99 - - [17/Sep/2024:00:20:29] "GET /metrics HTTP/1.1" 200 214911 "" "Prometheus/2.33.4" 2024-09-17T00:20:29.753 INFO:teuthology.orchestra.run.smithi002.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-17T00:20:29.753 INFO:teuthology.orchestra.run.smithi002.stdout:alertmanager.a smithi002 *:9093,9094 running (2m) 2m ago 3m 17.4M - ba2b418f427c f8bd5774d324 2024-09-17T00:20:29.753 INFO:teuthology.orchestra.run.smithi002.stdout:grafana.a smithi099 *:3000 running (3m) 2m ago 3m 47.1M - 8.3.5 dad864ee21e9 1547cd171832 2024-09-17T00:20:29.753 INFO:teuthology.orchestra.run.smithi002.stdout:iscsi.foo.smithi002.vmouem smithi002 running (2m) 2m ago 2m 59.6M - 3.5 e1d6a67b021e 6b47041ec00f 2024-09-17T00:20:29.753 INFO:teuthology.orchestra.run.smithi002.stdout:mgr.x smithi099 *:8443 running (7m) 2m ago 7m 411M - 17.2.0 e1d6a67b021e 9db4cd4717a5 2024-09-17T00:20:29.753 INFO:teuthology.orchestra.run.smithi002.stdout:mgr.y smithi002 *:9283 running (9m) 2m ago 9m 455M - 17.2.0 e1d6a67b021e d32b04d294d3 2024-09-17T00:20:29.753 INFO:teuthology.orchestra.run.smithi002.stdout:mon.a smithi002 running (9m) 2m ago 9m 46.4M 2048M 17.2.0 e1d6a67b021e a68823df625f 2024-09-17T00:20:29.754 INFO:teuthology.orchestra.run.smithi002.stdout:mon.b smithi099 running (8m) 2m ago 8m 40.6M 2048M 17.2.0 e1d6a67b021e c6d2af715187 2024-09-17T00:20:29.754 INFO:teuthology.orchestra.run.smithi002.stdout:mon.c smithi002 running (8m) 2m ago 8m 39.2M 2048M 17.2.0 e1d6a67b021e 77b3d2e14899 2024-09-17T00:20:29.754 INFO:teuthology.orchestra.run.smithi002.stdout:node-exporter.a smithi002 *:9100 running (4m) 2m ago 3m 17.3M - 1dbe0e931976 d28913b4af30 2024-09-17T00:20:29.754 INFO:teuthology.orchestra.run.smithi002.stdout:node-exporter.b smithi099 *:9100 running (3m) 2m ago 3m 15.9M - 1dbe0e931976 352ae73fe2a7 2024-09-17T00:20:29.754 INFO:teuthology.orchestra.run.smithi002.stdout:osd.0 smithi002 running (7m) 2m ago 7m 49.5M 4017M 17.2.0 e1d6a67b021e a0ad0fb7aa34 2024-09-17T00:20:29.754 INFO:teuthology.orchestra.run.smithi002.stdout:osd.1 smithi002 running (7m) 2m ago 7m 51.4M 4017M 17.2.0 e1d6a67b021e 7cb62c311c7d 2024-09-17T00:20:29.754 INFO:teuthology.orchestra.run.smithi002.stdout:osd.2 smithi002 running (6m) 2m ago 6m 49.1M 4017M 17.2.0 e1d6a67b021e eafddacb955e 2024-09-17T00:20:29.754 INFO:teuthology.orchestra.run.smithi002.stdout:osd.3 smithi002 running (6m) 2m ago 6m 51.8M 4017M 17.2.0 e1d6a67b021e fd99d3764412 2024-09-17T00:20:29.754 INFO:teuthology.orchestra.run.smithi002.stdout:osd.4 smithi099 running (5m) 2m ago 5m 51.5M 4272M 17.2.0 e1d6a67b021e 0af1827b658d 2024-09-17T00:20:29.754 INFO:teuthology.orchestra.run.smithi002.stdout:osd.5 smithi099 running (5m) 2m ago 5m 50.0M 4272M 17.2.0 e1d6a67b021e 4f7ef3aefc58 2024-09-17T00:20:29.754 INFO:teuthology.orchestra.run.smithi002.stdout:osd.6 smithi099 running (5m) 2m ago 5m 45.3M 4272M 17.2.0 e1d6a67b021e 3213f5e6952a 2024-09-17T00:20:29.754 INFO:teuthology.orchestra.run.smithi002.stdout:osd.7 smithi099 running (4m) 2m ago 4m 51.6M 4272M 17.2.0 e1d6a67b021e 63be3a1ed0a1 2024-09-17T00:20:29.754 INFO:teuthology.orchestra.run.smithi002.stdout:prometheus.a smithi099 *:9095 running (2m) 2m ago 4m 44.5M - 514e6a882f6e a1529c2c7a1c 2024-09-17T00:20:29.754 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.foo.smithi002.uwdhcs smithi002 *:8000 running (2m) 2m ago 2m 71.2M - 17.2.0 e1d6a67b021e 9c35813f459a 2024-09-17T00:20:29.755 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.foo.smithi099.utzfap smithi099 *:8000 running (2m) 2m ago 2m 71.1M - 17.2.0 e1d6a67b021e 34ec989160a6 2024-09-17T00:20:29.755 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.smpl.smithi002.rqnxii smithi002 *:80 running (2m) 2m ago 2m 70.2M - 17.2.0 e1d6a67b021e 0e180d0afb99 2024-09-17T00:20:29.755 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.smpl.smithi099.nzvkpm smithi099 *:80 running (2m) 2m ago 2m 71.5M - 17.2.0 e1d6a67b021e 6a068e82ac72 2024-09-17T00:20:30.259 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-09-17T00:20:30.259 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": { 2024-09-17T00:20:30.260 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-09-17T00:20:30.260 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-09-17T00:20:30.260 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": { 2024-09-17T00:20:30.260 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-09-17T00:20:30.260 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-09-17T00:20:30.260 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": { 2024-09-17T00:20:30.260 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-09-17T00:20:30.260 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-09-17T00:20:30.260 INFO:teuthology.orchestra.run.smithi002.stdout: "mds": {}, 2024-09-17T00:20:30.260 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw": { 2024-09-17T00:20:30.260 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-09-17T00:20:30.261 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-09-17T00:20:30.261 INFO:teuthology.orchestra.run.smithi002.stdout: "overall": { 2024-09-17T00:20:30.261 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 17 2024-09-17T00:20:30.261 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-09-17T00:20:30.261 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-09-17T00:20:30.466 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:20:30 smithi099 ceph-mon[24594]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:20:30.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:20:30 smithi099 ceph-mon[24594]: from='client.15051 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:20:30.478 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:20:30 smithi002 ceph-mon[28203]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:20:30.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:20:30 smithi002 ceph-mon[28203]: from='client.15051 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:20:30.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:20:30 smithi002 ceph-mon[23737]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:20:30.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:20:30 smithi002 ceph-mon[23737]: from='client.15051 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:20:30.695 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-09-17T00:20:30.695 INFO:teuthology.orchestra.run.smithi002.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f", 2024-09-17T00:20:30.695 INFO:teuthology.orchestra.run.smithi002.stdout: "in_progress": true, 2024-09-17T00:20:30.695 INFO:teuthology.orchestra.run.smithi002.stdout: "services_complete": [], 2024-09-17T00:20:30.695 INFO:teuthology.orchestra.run.smithi002.stdout: "progress": "", 2024-09-17T00:20:30.695 INFO:teuthology.orchestra.run.smithi002.stdout: "message": "Doing first pull of quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f image" 2024-09-17T00:20:30.695 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-09-17T00:20:31.133 INFO:teuthology.orchestra.run.smithi002.stdout:HEALTH_OK 2024-09-17T00:20:31.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:20:31 smithi099 ceph-mon[24594]: from='client.15057 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:20:31.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:20:31 smithi099 ceph-mon[24594]: from='client.15063 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:20:31.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:20:31 smithi099 ceph-mon[24594]: from='client.? 172.21.15.2:0/1565585203' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:20:31.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:20:31 smithi099 ceph-mon[24594]: from='client.? 172.21.15.2:0/1129194019' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T00:20:31.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:20:31 smithi002 ceph-mon[28203]: from='client.15057 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:20:31.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:20:31 smithi002 ceph-mon[28203]: from='client.15063 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:20:31.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:20:31 smithi002 ceph-mon[28203]: from='client.? 172.21.15.2:0/1565585203' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:20:31.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:20:31 smithi002 ceph-mon[28203]: from='client.? 172.21.15.2:0/1129194019' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T00:20:31.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:20:31 smithi002 ceph-mon[23737]: from='client.15057 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:20:31.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:20:31 smithi002 ceph-mon[23737]: from='client.15063 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:20:31.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:20:31 smithi002 ceph-mon[23737]: from='client.? 172.21.15.2:0/1565585203' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:20:31.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:20:31 smithi002 ceph-mon[23737]: from='client.? 172.21.15.2:0/1129194019' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T00:20:32.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:20:32 smithi099 ceph-mon[24594]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:20:32.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:20:32 smithi099 ceph-mon[24594]: from='client.15075 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:20:32.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:20:32 smithi002 ceph-mon[23737]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:20:32.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:20:32 smithi002 ceph-mon[23737]: from='client.15075 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:20:32.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:20:32 smithi002 ceph-mon[28203]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:20:32.480 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:20:32 smithi002 ceph-mon[28203]: from='client.15075 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:20:34.466 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:20:34 smithi099 ceph-mon[24594]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:20:34.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:20:34 smithi099 ceph-mon[24594]: from='client.14889 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:20:34.478 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:20:34 smithi002 ceph-mon[23737]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:20:34.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:20:34 smithi002 ceph-mon[23737]: from='client.14889 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:20:34.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:20:34 smithi002 ceph-mon[28203]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:20:34.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:20:34 smithi002 ceph-mon[28203]: from='client.14889 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:20:35.216 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:20:34 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[25772]: ::ffff:172.21.15.99 - - [17/Sep/2024:00:20:34] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-09-17T00:20:36.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:20:36 smithi099 ceph-mon[24594]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:20:36.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:20:36 smithi002 ceph-mon[23737]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:20:36.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:20:36 smithi002 ceph-mon[28203]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:20:38.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:20:38 smithi099 ceph-mon[24594]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:20:38.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:20:38 smithi002 ceph-mon[28203]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:20:38.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:20:38 smithi002 ceph-mon[23737]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:20:39.478 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:20:39 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: ::ffff:172.21.15.99 - - [17/Sep/2024:00:20:39] "GET /metrics HTTP/1.1" 200 214911 "" "Prometheus/2.33.4" 2024-09-17T00:20:40.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:20:40 smithi099 ceph-mon[24594]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:20:40.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:20:40 smithi002 ceph-mon[28203]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:20:40.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:20:40 smithi002 ceph-mon[23737]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:20:42.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:20:42 smithi099 ceph-mon[24594]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:20:42.478 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:20:42 smithi002 ceph-mon[28203]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:20:42.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:20:42 smithi002 ceph-mon[23737]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:20:44.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:20:44 smithi099 ceph-mon[24594]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:20:44.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:20:44 smithi099 ceph-mon[24594]: from='client.14889 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:20:44.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:20:44 smithi002 ceph-mon[28203]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:20:44.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:20:44 smithi002 ceph-mon[28203]: from='client.14889 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:20:44.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:20:44 smithi002 ceph-mon[23737]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:20:44.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:20:44 smithi002 ceph-mon[23737]: from='client.14889 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:20:45.217 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:20:44 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[25772]: ::ffff:172.21.15.99 - - [17/Sep/2024:00:20:44] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-09-17T00:20:46.466 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:20:46 smithi099 ceph-mon[24594]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:20:46.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:20:46 smithi002 ceph-mon[28203]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:20:46.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:20:46 smithi002 ceph-mon[23737]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:20:48.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:20:48 smithi099 ceph-mon[24594]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:20:48.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:20:48 smithi002 ceph-mon[28203]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:20:48.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:20:48 smithi002 ceph-mon[23737]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:20:49.479 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:20:49 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: ::ffff:172.21.15.99 - - [17/Sep/2024:00:20:49] "GET /metrics HTTP/1.1" 200 214886 "" "Prometheus/2.33.4" 2024-09-17T00:20:50.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:20:50 smithi099 ceph-mon[24594]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:20:50.478 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:20:50 smithi002 ceph-mon[23737]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:20:50.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:20:50 smithi002 ceph-mon[28203]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:20:52.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:20:52 smithi099 ceph-mon[24594]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:20:52.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:20:52 smithi002 ceph-mon[23737]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:20:52.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:20:52 smithi002 ceph-mon[28203]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:20:54.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:20:54 smithi099 ceph-mon[24594]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:20:54.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:20:54 smithi099 ceph-mon[24594]: from='client.14889 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:20:54.478 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:20:54 smithi002 ceph-mon[28203]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:20:54.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:20:54 smithi002 ceph-mon[28203]: from='client.14889 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:20:54.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:20:54 smithi002 ceph-mon[23737]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:20:54.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:20:54 smithi002 ceph-mon[23737]: from='client.14889 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:20:55.217 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:20:54 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[25772]: ::ffff:172.21.15.99 - - [17/Sep/2024:00:20:54] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-09-17T00:20:56.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:20:56 smithi099 ceph-mon[24594]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:20:56.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:20:56 smithi002 ceph-mon[28203]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:20:56.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:20:56 smithi002 ceph-mon[23737]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:20:58.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:20:58 smithi099 ceph-mon[24594]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:20:58.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:20:58 smithi002 ceph-mon[28203]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:20:58.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:20:58 smithi002 ceph-mon[23737]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:20:59.479 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:20:59 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: ::ffff:172.21.15.99 - - [17/Sep/2024:00:20:59] "GET /metrics HTTP/1.1" 200 214857 "" "Prometheus/2.33.4" 2024-09-17T00:21:00.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:00 smithi099 ceph-mon[24594]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:21:00.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:00 smithi002 ceph-mon[23737]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:21:00.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:00 smithi002 ceph-mon[28203]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:21:01.488 INFO:teuthology.orchestra.run.smithi002.stdout:true 2024-09-17T00:21:02.112 INFO:teuthology.orchestra.run.smithi002.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-17T00:21:02.113 INFO:teuthology.orchestra.run.smithi002.stdout:alertmanager.a smithi002 *:9093,9094 running (3m) 2m ago 3m 17.4M - ba2b418f427c f8bd5774d324 2024-09-17T00:21:02.113 INFO:teuthology.orchestra.run.smithi002.stdout:grafana.a smithi099 *:3000 running (3m) 2m ago 3m 47.1M - 8.3.5 dad864ee21e9 1547cd171832 2024-09-17T00:21:02.113 INFO:teuthology.orchestra.run.smithi002.stdout:iscsi.foo.smithi002.vmouem smithi002 running (2m) 2m ago 2m 59.6M - 3.5 e1d6a67b021e 6b47041ec00f 2024-09-17T00:21:02.113 INFO:teuthology.orchestra.run.smithi002.stdout:mgr.x smithi099 *:8443 running (8m) 2m ago 8m 411M - 17.2.0 e1d6a67b021e 9db4cd4717a5 2024-09-17T00:21:02.113 INFO:teuthology.orchestra.run.smithi002.stdout:mgr.y smithi002 *:9283 running (10m) 2m ago 10m 455M - 17.2.0 e1d6a67b021e d32b04d294d3 2024-09-17T00:21:02.113 INFO:teuthology.orchestra.run.smithi002.stdout:mon.a smithi002 running (10m) 2m ago 10m 46.4M 2048M 17.2.0 e1d6a67b021e a68823df625f 2024-09-17T00:21:02.113 INFO:teuthology.orchestra.run.smithi002.stdout:mon.b smithi099 running (8m) 2m ago 8m 40.6M 2048M 17.2.0 e1d6a67b021e c6d2af715187 2024-09-17T00:21:02.113 INFO:teuthology.orchestra.run.smithi002.stdout:mon.c smithi002 running (8m) 2m ago 8m 39.2M 2048M 17.2.0 e1d6a67b021e 77b3d2e14899 2024-09-17T00:21:02.113 INFO:teuthology.orchestra.run.smithi002.stdout:node-exporter.a smithi002 *:9100 running (4m) 2m ago 4m 17.3M - 1dbe0e931976 d28913b4af30 2024-09-17T00:21:02.113 INFO:teuthology.orchestra.run.smithi002.stdout:node-exporter.b smithi099 *:9100 running (4m) 2m ago 4m 15.9M - 1dbe0e931976 352ae73fe2a7 2024-09-17T00:21:02.114 INFO:teuthology.orchestra.run.smithi002.stdout:osd.0 smithi002 running (8m) 2m ago 8m 49.5M 4017M 17.2.0 e1d6a67b021e a0ad0fb7aa34 2024-09-17T00:21:02.114 INFO:teuthology.orchestra.run.smithi002.stdout:osd.1 smithi002 running (7m) 2m ago 7m 51.4M 4017M 17.2.0 e1d6a67b021e 7cb62c311c7d 2024-09-17T00:21:02.114 INFO:teuthology.orchestra.run.smithi002.stdout:osd.2 smithi002 running (7m) 2m ago 7m 49.1M 4017M 17.2.0 e1d6a67b021e eafddacb955e 2024-09-17T00:21:02.114 INFO:teuthology.orchestra.run.smithi002.stdout:osd.3 smithi002 running (6m) 2m ago 6m 51.8M 4017M 17.2.0 e1d6a67b021e fd99d3764412 2024-09-17T00:21:02.114 INFO:teuthology.orchestra.run.smithi002.stdout:osd.4 smithi099 running (6m) 2m ago 6m 51.5M 4272M 17.2.0 e1d6a67b021e 0af1827b658d 2024-09-17T00:21:02.114 INFO:teuthology.orchestra.run.smithi002.stdout:osd.5 smithi099 running (6m) 2m ago 6m 50.0M 4272M 17.2.0 e1d6a67b021e 4f7ef3aefc58 2024-09-17T00:21:02.114 INFO:teuthology.orchestra.run.smithi002.stdout:osd.6 smithi099 running (5m) 2m ago 5m 45.3M 4272M 17.2.0 e1d6a67b021e 3213f5e6952a 2024-09-17T00:21:02.114 INFO:teuthology.orchestra.run.smithi002.stdout:osd.7 smithi099 running (5m) 2m ago 5m 51.6M 4272M 17.2.0 e1d6a67b021e 63be3a1ed0a1 2024-09-17T00:21:02.114 INFO:teuthology.orchestra.run.smithi002.stdout:prometheus.a smithi099 *:9095 running (3m) 2m ago 4m 44.5M - 514e6a882f6e a1529c2c7a1c 2024-09-17T00:21:02.114 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.foo.smithi002.uwdhcs smithi002 *:8000 running (3m) 2m ago 3m 71.2M - 17.2.0 e1d6a67b021e 9c35813f459a 2024-09-17T00:21:02.114 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.foo.smithi099.utzfap smithi099 *:8000 running (3m) 2m ago 2m 71.1M - 17.2.0 e1d6a67b021e 34ec989160a6 2024-09-17T00:21:02.114 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.smpl.smithi002.rqnxii smithi002 *:80 running (2m) 2m ago 2m 70.2M - 17.2.0 e1d6a67b021e 0e180d0afb99 2024-09-17T00:21:02.114 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.smpl.smithi099.nzvkpm smithi099 *:80 running (2m) 2m ago 2m 71.5M - 17.2.0 e1d6a67b021e 6a068e82ac72 2024-09-17T00:21:02.407 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:02 smithi002 ceph-mon[23737]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:21:02.407 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:02 smithi002 ceph-mon[28203]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:21:02.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:02 smithi099 ceph-mon[24594]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:21:02.483 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-09-17T00:21:02.483 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": { 2024-09-17T00:21:02.483 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-09-17T00:21:02.483 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-09-17T00:21:02.484 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": { 2024-09-17T00:21:02.484 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-09-17T00:21:02.484 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-09-17T00:21:02.484 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": { 2024-09-17T00:21:02.484 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-09-17T00:21:02.484 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-09-17T00:21:02.484 INFO:teuthology.orchestra.run.smithi002.stdout: "mds": {}, 2024-09-17T00:21:02.484 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw": { 2024-09-17T00:21:02.484 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-09-17T00:21:02.484 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-09-17T00:21:02.484 INFO:teuthology.orchestra.run.smithi002.stdout: "overall": { 2024-09-17T00:21:02.484 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 17 2024-09-17T00:21:02.484 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-09-17T00:21:02.484 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-09-17T00:21:02.805 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-09-17T00:21:02.805 INFO:teuthology.orchestra.run.smithi002.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f", 2024-09-17T00:21:02.805 INFO:teuthology.orchestra.run.smithi002.stdout: "in_progress": true, 2024-09-17T00:21:02.805 INFO:teuthology.orchestra.run.smithi002.stdout: "services_complete": [], 2024-09-17T00:21:02.805 INFO:teuthology.orchestra.run.smithi002.stdout: "progress": "0/23 daemons upgraded", 2024-09-17T00:21:02.805 INFO:teuthology.orchestra.run.smithi002.stdout: "message": "Pulling quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f image on host smithi099" 2024-09-17T00:21:02.805 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-09-17T00:21:03.171 INFO:teuthology.orchestra.run.smithi002.stdout:HEALTH_OK 2024-09-17T00:21:03.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:03 smithi099 ceph-mon[24594]: from='client.24976 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:21:03.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:03 smithi099 ceph-mon[24594]: from='client.15093 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:21:03.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:03 smithi099 ceph-mon[24594]: from='client.24988 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:21:03.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:03 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:21:03.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:03 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T00:21:03.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:03 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:21:03.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:03 smithi099 ceph-mon[24594]: from='client.? 172.21.15.2:0/548575635' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:21:03.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:03 smithi099 ceph-mon[24594]: from='client.? 172.21.15.2:0/1687062849' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T00:21:03.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:03 smithi002 ceph-mon[28203]: from='client.24976 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:21:03.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:03 smithi002 ceph-mon[28203]: from='client.15093 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:21:03.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:03 smithi002 ceph-mon[28203]: from='client.24988 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:21:03.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:03 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:21:03.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:03 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T00:21:03.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:03 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:21:03.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:03 smithi002 ceph-mon[28203]: from='client.? 172.21.15.2:0/548575635' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:21:03.480 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:03 smithi002 ceph-mon[28203]: from='client.? 172.21.15.2:0/1687062849' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T00:21:03.480 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:03 smithi002 ceph-mon[23737]: from='client.24976 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:21:03.480 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:03 smithi002 ceph-mon[23737]: from='client.15093 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:21:03.480 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:03 smithi002 ceph-mon[23737]: from='client.24988 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:21:03.480 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:03 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:21:03.480 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:03 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T00:21:03.480 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:03 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:21:03.480 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:03 smithi002 ceph-mon[23737]: from='client.? 172.21.15.2:0/548575635' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:21:03.480 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:03 smithi002 ceph-mon[23737]: from='client.? 172.21.15.2:0/1687062849' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T00:21:03.979 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Sep 17 00:21:03 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-alertmanager-a[48044]: level=warn ts=2024-09-17T00:21:03.507Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.2:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.2 because it doesn't contain any IP SANs" 2024-09-17T00:21:03.979 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Sep 17 00:21:03 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-alertmanager-a[48044]: level=warn ts=2024-09-17T00:21:03.507Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.99:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.99 because it doesn't contain any IP SANs" 2024-09-17T00:21:04.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:04 smithi099 ceph-mon[24594]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:21:04.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:04 smithi099 ceph-mon[24594]: Upgrade: Target is version 19.1.1-330-gc2ddf972 (unknown) 2024-09-17T00:21:04.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:04 smithi099 ceph-mon[24594]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, digests ['quay.ceph.io/ceph-ci/ceph@sha256:475e68c989240ebfb6ca26a687a713828159881ccf6edcf69b7320f1853af3e3'] 2024-09-17T00:21:04.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:04 smithi099 ceph-mon[24594]: Upgrade: Need to upgrade myself (mgr.y) 2024-09-17T00:21:04.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:04 smithi099 ceph-mon[24594]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f on smithi099 2024-09-17T00:21:04.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:04 smithi099 ceph-mon[24594]: from='client.25000 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:21:04.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:04 smithi099 ceph-mon[24594]: from='client.14889 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:21:04.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:04 smithi002 ceph-mon[28203]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:21:04.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:04 smithi002 ceph-mon[28203]: Upgrade: Target is version 19.1.1-330-gc2ddf972 (unknown) 2024-09-17T00:21:04.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:04 smithi002 ceph-mon[28203]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, digests ['quay.ceph.io/ceph-ci/ceph@sha256:475e68c989240ebfb6ca26a687a713828159881ccf6edcf69b7320f1853af3e3'] 2024-09-17T00:21:04.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:04 smithi002 ceph-mon[28203]: Upgrade: Need to upgrade myself (mgr.y) 2024-09-17T00:21:04.480 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:04 smithi002 ceph-mon[28203]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f on smithi099 2024-09-17T00:21:04.480 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:04 smithi002 ceph-mon[28203]: from='client.25000 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:21:04.480 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:04 smithi002 ceph-mon[28203]: from='client.14889 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:21:04.480 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:04 smithi002 ceph-mon[23737]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:21:04.480 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:04 smithi002 ceph-mon[23737]: Upgrade: Target is version 19.1.1-330-gc2ddf972 (unknown) 2024-09-17T00:21:04.480 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:04 smithi002 ceph-mon[23737]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, digests ['quay.ceph.io/ceph-ci/ceph@sha256:475e68c989240ebfb6ca26a687a713828159881ccf6edcf69b7320f1853af3e3'] 2024-09-17T00:21:04.481 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:04 smithi002 ceph-mon[23737]: Upgrade: Need to upgrade myself (mgr.y) 2024-09-17T00:21:04.481 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:04 smithi002 ceph-mon[23737]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f on smithi099 2024-09-17T00:21:04.481 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:04 smithi002 ceph-mon[23737]: from='client.25000 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:21:04.481 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:04 smithi002 ceph-mon[23737]: from='client.14889 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:21:05.216 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:21:04 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[25772]: ::ffff:172.21.15.99 - - [17/Sep/2024:00:21:04] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-09-17T00:21:06.466 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:06 smithi099 ceph-mon[24594]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:21:06.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:06 smithi002 ceph-mon[23737]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:21:06.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:06 smithi002 ceph-mon[28203]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:21:08.466 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:08 smithi099 ceph-mon[24594]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:21:08.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:08 smithi002 ceph-mon[28203]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:21:08.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:08 smithi002 ceph-mon[23737]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:21:09.480 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:21:09 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: ::ffff:172.21.15.99 - - [17/Sep/2024:00:21:09] "GET /metrics HTTP/1.1" 200 214857 "" "Prometheus/2.33.4" 2024-09-17T00:21:10.466 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:10 smithi099 ceph-mon[24594]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:21:10.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:10 smithi002 ceph-mon[23737]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:21:10.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:10 smithi002 ceph-mon[28203]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:21:11.468 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:11 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T00:21:11.468 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:11 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T00:21:11.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:11 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T00:21:11.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:11 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T00:21:11.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:11 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T00:21:11.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:11 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T00:21:12.466 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:12 smithi099 ceph-mon[24594]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:21:12.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:12 smithi002 ceph-mon[23737]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:21:12.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:12 smithi002 ceph-mon[28203]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:21:13.979 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Sep 17 00:21:13 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-alertmanager-a[48044]: level=error ts=2024-09-17T00:21:13.505Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.99:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.99 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.2:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.2 because it doesn't contain any IP SANs" 2024-09-17T00:21:13.979 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Sep 17 00:21:13 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-alertmanager-a[48044]: level=warn ts=2024-09-17T00:21:13.507Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.99:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.99 because it doesn't contain any IP SANs" 2024-09-17T00:21:13.979 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Sep 17 00:21:13 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-alertmanager-a[48044]: level=warn ts=2024-09-17T00:21:13.508Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.2:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.2 because it doesn't contain any IP SANs" 2024-09-17T00:21:14.466 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:14 smithi099 ceph-mon[24594]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:21:14.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:14 smithi099 ceph-mon[24594]: from='client.14889 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:21:14.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:14 smithi002 ceph-mon[28203]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:21:14.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:14 smithi002 ceph-mon[28203]: from='client.14889 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:21:14.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:14 smithi002 ceph-mon[23737]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:21:14.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:14 smithi002 ceph-mon[23737]: from='client.14889 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:21:15.216 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:21:14 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[25772]: ::ffff:172.21.15.99 - - [17/Sep/2024:00:21:14] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-09-17T00:21:16.466 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:16 smithi099 ceph-mon[24594]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:21:16.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:16 smithi002 ceph-mon[23737]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:21:16.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:16 smithi002 ceph-mon[28203]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:21:18.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:18 smithi002 ceph-mon[23737]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:21:18.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:18 smithi002 ceph-mon[28203]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:21:18.716 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:18 smithi099 ceph-mon[24594]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:21:19.479 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:21:19 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: ::ffff:172.21.15.99 - - [17/Sep/2024:00:21:19] "GET /metrics HTTP/1.1" 200 214861 "" "Prometheus/2.33.4" 2024-09-17T00:21:20.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:20 smithi002 ceph-mon[23737]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:21:20.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:20 smithi002 ceph-mon[28203]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:21:20.716 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:20 smithi099 ceph-mon[24594]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:21:22.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:22 smithi002 ceph-mon[23737]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:21:22.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:22 smithi002 ceph-mon[28203]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:21:22.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:22 smithi099 ceph-mon[24594]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:21:23.979 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Sep 17 00:21:23 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-alertmanager-a[48044]: level=error ts=2024-09-17T00:21:23.506Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.2:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.2 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.99:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.99 because it doesn't contain any IP SANs" 2024-09-17T00:21:23.979 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Sep 17 00:21:23 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-alertmanager-a[48044]: level=warn ts=2024-09-17T00:21:23.508Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.2:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.2 because it doesn't contain any IP SANs" 2024-09-17T00:21:23.979 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Sep 17 00:21:23 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-alertmanager-a[48044]: level=warn ts=2024-09-17T00:21:23.508Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.99:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.99 because it doesn't contain any IP SANs" 2024-09-17T00:21:24.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:24 smithi002 ceph-mon[23737]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:21:24.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:24 smithi002 ceph-mon[23737]: from='client.14889 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:21:24.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:24 smithi002 ceph-mon[28203]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:21:24.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:24 smithi002 ceph-mon[28203]: from='client.14889 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:21:24.716 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:24 smithi099 ceph-mon[24594]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:21:24.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:24 smithi099 ceph-mon[24594]: from='client.14889 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:21:25.216 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:21:24 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[25772]: ::ffff:172.21.15.99 - - [17/Sep/2024:00:21:24] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-09-17T00:21:26.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:26 smithi002 ceph-mon[28203]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:21:26.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:26 smithi002 ceph-mon[23737]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:21:26.506 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:26 smithi099 ceph-mon[24594]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:21:28.716 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:28 smithi099 ceph-mon[24594]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:21:28.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:28 smithi002 ceph-mon[28203]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:21:28.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:28 smithi002 ceph-mon[23737]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:21:29.479 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:21:29 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: ::ffff:172.21.15.99 - - [17/Sep/2024:00:21:29] "GET /metrics HTTP/1.1" 200 214851 "" "Prometheus/2.33.4" 2024-09-17T00:21:30.716 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:30 smithi099 ceph-mon[24594]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:21:30.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:30 smithi002 ceph-mon[28203]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:21:30.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:30 smithi002 ceph-mon[23737]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:21:32.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:32 smithi099 ceph-mon[24594]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:21:32.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:32 smithi002 ceph-mon[28203]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:21:32.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:32 smithi002 ceph-mon[23737]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:21:33.511 INFO:teuthology.orchestra.run.smithi002.stdout:true 2024-09-17T00:21:33.926 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Sep 17 00:21:33 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-alertmanager-a[48044]: level=error ts=2024-09-17T00:21:33.507Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.99:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.99 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.2:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.2 because it doesn't contain any IP SANs" 2024-09-17T00:21:33.926 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Sep 17 00:21:33 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-alertmanager-a[48044]: level=warn ts=2024-09-17T00:21:33.508Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.2:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.2 because it doesn't contain any IP SANs" 2024-09-17T00:21:33.926 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Sep 17 00:21:33 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-alertmanager-a[48044]: level=warn ts=2024-09-17T00:21:33.509Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.99:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.99 because it doesn't contain any IP SANs" 2024-09-17T00:21:34.120 INFO:teuthology.orchestra.run.smithi002.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-17T00:21:34.120 INFO:teuthology.orchestra.run.smithi002.stdout:alertmanager.a smithi002 *:9093,9094 running (3m) 3m ago 4m 17.4M - ba2b418f427c f8bd5774d324 2024-09-17T00:21:34.120 INFO:teuthology.orchestra.run.smithi002.stdout:grafana.a smithi099 *:3000 running (4m) 3m ago 4m 47.1M - 8.3.5 dad864ee21e9 1547cd171832 2024-09-17T00:21:34.120 INFO:teuthology.orchestra.run.smithi002.stdout:iscsi.foo.smithi002.vmouem smithi002 running (3m) 3m ago 3m 59.6M - 3.5 e1d6a67b021e 6b47041ec00f 2024-09-17T00:21:34.120 INFO:teuthology.orchestra.run.smithi002.stdout:mgr.x smithi099 *:8443 running (8m) 3m ago 8m 411M - 17.2.0 e1d6a67b021e 9db4cd4717a5 2024-09-17T00:21:34.120 INFO:teuthology.orchestra.run.smithi002.stdout:mgr.y smithi002 *:9283 running (10m) 3m ago 10m 455M - 17.2.0 e1d6a67b021e d32b04d294d3 2024-09-17T00:21:34.121 INFO:teuthology.orchestra.run.smithi002.stdout:mon.a smithi002 running (10m) 3m ago 10m 46.4M 2048M 17.2.0 e1d6a67b021e a68823df625f 2024-09-17T00:21:34.121 INFO:teuthology.orchestra.run.smithi002.stdout:mon.b smithi099 running (9m) 3m ago 9m 40.6M 2048M 17.2.0 e1d6a67b021e c6d2af715187 2024-09-17T00:21:34.121 INFO:teuthology.orchestra.run.smithi002.stdout:mon.c smithi002 running (9m) 3m ago 9m 39.2M 2048M 17.2.0 e1d6a67b021e 77b3d2e14899 2024-09-17T00:21:34.121 INFO:teuthology.orchestra.run.smithi002.stdout:node-exporter.a smithi002 *:9100 running (5m) 3m ago 5m 17.3M - 1dbe0e931976 d28913b4af30 2024-09-17T00:21:34.121 INFO:teuthology.orchestra.run.smithi002.stdout:node-exporter.b smithi099 *:9100 running (5m) 3m ago 5m 15.9M - 1dbe0e931976 352ae73fe2a7 2024-09-17T00:21:34.121 INFO:teuthology.orchestra.run.smithi002.stdout:osd.0 smithi002 running (8m) 3m ago 8m 49.5M 4017M 17.2.0 e1d6a67b021e a0ad0fb7aa34 2024-09-17T00:21:34.121 INFO:teuthology.orchestra.run.smithi002.stdout:osd.1 smithi002 running (8m) 3m ago 8m 51.4M 4017M 17.2.0 e1d6a67b021e 7cb62c311c7d 2024-09-17T00:21:34.121 INFO:teuthology.orchestra.run.smithi002.stdout:osd.2 smithi002 running (7m) 3m ago 7m 49.1M 4017M 17.2.0 e1d6a67b021e eafddacb955e 2024-09-17T00:21:34.121 INFO:teuthology.orchestra.run.smithi002.stdout:osd.3 smithi002 running (7m) 3m ago 7m 51.8M 4017M 17.2.0 e1d6a67b021e fd99d3764412 2024-09-17T00:21:34.121 INFO:teuthology.orchestra.run.smithi002.stdout:osd.4 smithi099 running (7m) 3m ago 7m 51.5M 4272M 17.2.0 e1d6a67b021e 0af1827b658d 2024-09-17T00:21:34.121 INFO:teuthology.orchestra.run.smithi002.stdout:osd.5 smithi099 running (6m) 3m ago 6m 50.0M 4272M 17.2.0 e1d6a67b021e 4f7ef3aefc58 2024-09-17T00:21:34.121 INFO:teuthology.orchestra.run.smithi002.stdout:osd.6 smithi099 running (6m) 3m ago 6m 45.3M 4272M 17.2.0 e1d6a67b021e 3213f5e6952a 2024-09-17T00:21:34.121 INFO:teuthology.orchestra.run.smithi002.stdout:osd.7 smithi099 running (5m) 3m ago 5m 51.6M 4272M 17.2.0 e1d6a67b021e 63be3a1ed0a1 2024-09-17T00:21:34.122 INFO:teuthology.orchestra.run.smithi002.stdout:prometheus.a smithi099 *:9095 running (3m) 3m ago 5m 44.5M - 514e6a882f6e a1529c2c7a1c 2024-09-17T00:21:34.122 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.foo.smithi002.uwdhcs smithi002 *:8000 running (3m) 3m ago 3m 71.2M - 17.2.0 e1d6a67b021e 9c35813f459a 2024-09-17T00:21:34.122 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.foo.smithi099.utzfap smithi099 *:8000 running (3m) 3m ago 3m 71.1M - 17.2.0 e1d6a67b021e 34ec989160a6 2024-09-17T00:21:34.122 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.smpl.smithi002.rqnxii smithi002 *:80 running (3m) 3m ago 3m 70.2M - 17.2.0 e1d6a67b021e 0e180d0afb99 2024-09-17T00:21:34.122 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.smpl.smithi099.nzvkpm smithi099 *:80 running (3m) 3m ago 3m 71.5M - 17.2.0 e1d6a67b021e 6a068e82ac72 2024-09-17T00:21:34.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:33 smithi099 ceph-mon[24594]: pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:21:34.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:33 smithi099 ceph-mon[24594]: from='client.14889 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:21:34.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:33 smithi099 ceph-mon[24594]: Upgrade: Updating mgr.x 2024-09-17T00:21:34.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:33 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:21:34.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:33 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-17T00:21:34.218 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:33 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-17T00:21:34.218 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:33 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:21:34.218 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:33 smithi099 ceph-mon[24594]: Deploying daemon mgr.x on smithi099 2024-09-17T00:21:34.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:33 smithi002 ceph-mon[23737]: pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:21:34.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:33 smithi002 ceph-mon[23737]: from='client.14889 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:21:34.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:33 smithi002 ceph-mon[23737]: Upgrade: Updating mgr.x 2024-09-17T00:21:34.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:33 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:21:34.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:33 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-17T00:21:34.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:33 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-17T00:21:34.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:33 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:21:34.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:33 smithi002 ceph-mon[23737]: Deploying daemon mgr.x on smithi099 2024-09-17T00:21:34.230 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:33 smithi002 ceph-mon[28203]: pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:21:34.230 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:33 smithi002 ceph-mon[28203]: from='client.14889 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:21:34.230 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:33 smithi002 ceph-mon[28203]: Upgrade: Updating mgr.x 2024-09-17T00:21:34.230 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:33 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:21:34.230 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:33 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-17T00:21:34.230 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:33 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-17T00:21:34.230 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:33 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:21:34.230 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:33 smithi002 ceph-mon[28203]: Deploying daemon mgr.x on smithi099 2024-09-17T00:21:34.485 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-09-17T00:21:34.485 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": { 2024-09-17T00:21:34.485 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-09-17T00:21:34.485 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-09-17T00:21:34.485 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": { 2024-09-17T00:21:34.486 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-09-17T00:21:34.486 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-09-17T00:21:34.486 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": { 2024-09-17T00:21:34.486 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-09-17T00:21:34.486 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-09-17T00:21:34.486 INFO:teuthology.orchestra.run.smithi002.stdout: "mds": {}, 2024-09-17T00:21:34.486 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw": { 2024-09-17T00:21:34.486 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-09-17T00:21:34.486 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-09-17T00:21:34.486 INFO:teuthology.orchestra.run.smithi002.stdout: "overall": { 2024-09-17T00:21:34.486 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 17 2024-09-17T00:21:34.486 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-09-17T00:21:34.486 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-09-17T00:21:34.697 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:21:34 smithi099 systemd[1]: Stopping Ceph mgr.x for 3aad27a2-7489-11ef-bceb-c7b262605968... 2024-09-17T00:21:34.697 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:21:34 smithi099 bash[45505]: Error: no container with name or ID "ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr.x" found: no such container 2024-09-17T00:21:34.697 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:21:34 smithi099 podman[45516]: 2024-09-17 00:21:34.453477233 +0000 UTC m=+0.137999245 container died 9db4cd4717a560ad4ae1fd9ae5446c6a9198d440b7ac1a04807a7d4946f8b47e (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, RELEASE=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, CEPH_POINT_RELEASE=-17.2.0, io.openshift.expose-services=, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, build-date=2022-05-03T08:36:31.336870, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.tags=base centos centos-stream, GIT_BRANCH=HEAD, io.buildah.version=1.19.8, maintainer=Guillaume Abrioux , vcs-type=git, vendor=Red Hat, Inc., release=754, ceph=True, version=8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_CLEAN=True, io.k8s.display-name=CentOS Stream 8, distribution-scope=public, architecture=x86_64, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.component=centos-stream-container, name=centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git) 2024-09-17T00:21:34.796 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-09-17T00:21:34.796 INFO:teuthology.orchestra.run.smithi002.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f", 2024-09-17T00:21:34.796 INFO:teuthology.orchestra.run.smithi002.stdout: "in_progress": true, 2024-09-17T00:21:34.796 INFO:teuthology.orchestra.run.smithi002.stdout: "services_complete": [], 2024-09-17T00:21:34.796 INFO:teuthology.orchestra.run.smithi002.stdout: "progress": "0/23 daemons upgraded", 2024-09-17T00:21:34.797 INFO:teuthology.orchestra.run.smithi002.stdout: "message": "Currently upgrading mgr daemons" 2024-09-17T00:21:34.797 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-09-17T00:21:34.797 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Sep 17 00:21:34 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-alertmanager-a[48044]: level=warn ts=2024-09-17T00:21:34.626Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=3 err="Post \"https://172.21.15.99:8443/api/prometheus_receiver\": dial tcp 172.21.15.99:8443: connect: connection refused" 2024-09-17T00:21:34.948 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:34 smithi099 ceph-mon[24594]: from='client.15123 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:21:34.948 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:34 smithi099 ceph-mon[24594]: from='client.25018 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:21:34.948 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:34 smithi099 ceph-mon[24594]: from='client.25024 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:21:34.948 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:34 smithi099 ceph-mon[24594]: from='client.? 172.21.15.2:0/2030685856' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:21:34.948 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:21:34 smithi099 podman[45516]: 2024-09-17 00:21:34.791319227 +0000 UTC m=+0.475841242 container cleanup 9db4cd4717a560ad4ae1fd9ae5446c6a9198d440b7ac1a04807a7d4946f8b47e (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, RELEASE=HEAD, distribution-scope=public, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=-17.2.0, io.buildah.version=1.19.8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, ceph=True, io.openshift.expose-services=, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.tags=base centos centos-stream, build-date=2022-05-03T08:36:31.336870, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-type=git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc., com.redhat.component=centos-stream-container, release=754, GIT_BRANCH=HEAD, architecture=x86_64, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=CentOS Stream 8, GIT_REPO=https://github.com/ceph/ceph-container.git, version=8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_CLEAN=True, name=centos-stream) 2024-09-17T00:21:34.949 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:21:34 smithi099 bash[45516]: ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x 2024-09-17T00:21:35.163 INFO:teuthology.orchestra.run.smithi002.stdout:HEALTH_OK 2024-09-17T00:21:35.163 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:34 smithi002 ceph-mon[23737]: from='client.15123 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:21:35.163 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:34 smithi002 ceph-mon[23737]: from='client.25018 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:21:35.163 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:34 smithi002 ceph-mon[23737]: from='client.25024 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:21:35.164 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:34 smithi002 ceph-mon[23737]: from='client.? 172.21.15.2:0/2030685856' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:21:35.164 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:34 smithi002 ceph-mon[28203]: from='client.15123 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:21:35.164 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:34 smithi002 ceph-mon[28203]: from='client.25018 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:21:35.164 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:34 smithi002 ceph-mon[28203]: from='client.25024 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:21:35.164 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:34 smithi002 ceph-mon[28203]: from='client.? 172.21.15.2:0/2030685856' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:21:35.207 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:21:34 smithi099 podman[45528]: 2024-09-17 00:21:34.951313907 +0000 UTC m=+0.491814491 container remove 9db4cd4717a560ad4ae1fd9ae5446c6a9198d440b7ac1a04807a7d4946f8b47e (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=-17.2.0, io.k8s.display-name=CentOS Stream 8, ceph=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, release=754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.buildah.version=1.19.8, GIT_CLEAN=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vcs-type=git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, distribution-scope=public, GIT_BRANCH=HEAD, build-date=2022-05-03T08:36:31.336870, vendor=Red Hat, Inc., RELEASE=HEAD, architecture=x86_64, com.redhat.component=centos-stream-container, name=centos-stream, version=8, io.openshift.tags=base centos centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image) 2024-09-17T00:21:35.208 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:21:34 smithi099 systemd[1]: ceph-3aad27a2-7489-11ef-bceb-c7b262605968@mgr.x.service: Main process exited, code=exited, status=143/n/a 2024-09-17T00:21:35.208 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:21:34 smithi099 bash[45543]: Error: no container with name or ID "ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr.x" found: no such container 2024-09-17T00:21:35.208 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:21:35 smithi099 systemd[1]: ceph-3aad27a2-7489-11ef-bceb-c7b262605968@mgr.x.service: Failed with result 'exit-code'. 2024-09-17T00:21:35.208 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:21:35 smithi099 systemd[1]: Stopped Ceph mgr.x for 3aad27a2-7489-11ef-bceb-c7b262605968. 2024-09-17T00:21:35.208 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:21:35 smithi099 systemd[1]: ceph-3aad27a2-7489-11ef-bceb-c7b262605968@mgr.x.service: Consumed 16.849s CPU time. 2024-09-17T00:21:35.466 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:21:35 smithi099 systemd[1]: Starting Ceph mgr.x for 3aad27a2-7489-11ef-bceb-c7b262605968... 2024-09-17T00:21:35.467 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:21:35 smithi099 podman[45641]: 2024-09-17 00:21:35.323756833 +0000 UTC m=+0.015960136 image pull ebb0615149def4a3179c82c0cc47af75808300abb9f1142788e5e605a7ac0e5b quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f 2024-09-17T00:21:35.467 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:21:35 smithi099 podman[45641]: 2024-09-17 00:21:35.461236483 +0000 UTC m=+0.153439777 container create a8e1e8964c84767502f0bfe594cd678554067c241ac97ddd9f857dc76d3f549e (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=squid-c2ddf97) 2024-09-17T00:21:35.840 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:21:35 smithi099 podman[45641]: 2024-09-17 00:21:35.6805231 +0000 UTC m=+0.372726409 container init a8e1e8964c84767502f0bfe594cd678554067c241ac97ddd9f857dc76d3f549e (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=squid-c2ddf97, GIT_BRANCH=HEAD) 2024-09-17T00:21:35.840 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:21:35 smithi099 podman[45641]: 2024-09-17 00:21:35.6832683 +0000 UTC m=+0.375471595 container start a8e1e8964c84767502f0bfe594cd678554067c241ac97ddd9f857dc76d3f549e (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x, RELEASE=squid-c2ddf97, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-09-17T00:21:35.840 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:21:35 smithi099 ceph-mgr[45660]: -- 172.21.15.99:0/4134411061 <== mon.2 v2:172.21.15.99:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55569d2f70e0 con 0x55569d2d4800 2024-09-17T00:21:35.840 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:21:35 smithi099 bash[45641]: a8e1e8964c84767502f0bfe594cd678554067c241ac97ddd9f857dc76d3f549e 2024-09-17T00:21:35.840 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:21:35 smithi099 systemd[1]: Started Ceph mgr.x for 3aad27a2-7489-11ef-bceb-c7b262605968. 2024-09-17T00:21:35.840 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:21:35 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: 2024-09-17T00:21:35.805+0000 7f0e25edf040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-17T00:21:36.103 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:35 smithi099 ceph-mon[24594]: pgmap v201: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:21:36.103 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:35 smithi099 ceph-mon[24594]: from='client.25036 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:21:36.103 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:35 smithi099 ceph-mon[24594]: from='client.? 172.21.15.2:0/319776920' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T00:21:36.103 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:35 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:21:36.104 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:35 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:21:36.104 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:35 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:21:36.104 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:21:35 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: 2024-09-17T00:21:35.869+0000 7f0e25edf040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-17T00:21:36.104 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:21:36 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: 2024-09-17T00:21:36.031+0000 7f0e25edf040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-17T00:21:36.104 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:21:36 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: 2024-09-17T00:21:36.105+0000 7f0e25edf040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-17T00:21:36.121 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:35 smithi002 ceph-mon[28203]: pgmap v201: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:21:36.121 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:35 smithi002 ceph-mon[28203]: from='client.25036 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:21:36.121 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:35 smithi002 ceph-mon[28203]: from='client.? 172.21.15.2:0/319776920' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T00:21:36.121 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:35 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:21:36.121 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:35 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:21:36.121 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:35 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:21:36.122 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:35 smithi002 ceph-mon[23737]: pgmap v201: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:21:36.122 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:35 smithi002 ceph-mon[23737]: from='client.25036 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:21:36.122 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:35 smithi002 ceph-mon[23737]: from='client.? 172.21.15.2:0/319776920' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T00:21:36.122 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:35 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:21:36.122 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:35 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:21:36.122 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:35 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:21:36.374 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:21:36 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: 2024-09-17T00:21:36.272+0000 7f0e25edf040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-17T00:21:36.716 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:21:36 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: 2024-09-17T00:21:36.418+0000 7f0e25edf040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-17T00:21:37.029 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:21:36 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: 2024-09-17T00:21:36.770+0000 7f0e25edf040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-17T00:21:37.029 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:21:36 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: 2024-09-17T00:21:36.833+0000 7f0e25edf040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-17T00:21:37.029 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:21:37 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: 2024-09-17T00:21:37.032+0000 7f0e25edf040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-17T00:21:37.326 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:37 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:21:37.354 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:37 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:21:37.354 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:37 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:21:38.102 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:21:37 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: 2024-09-17T00:21:37.988+0000 7f0e25edf040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-17T00:21:38.103 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:21:38 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: 2024-09-17T00:21:38.068+0000 7f0e25edf040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-17T00:21:38.466 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:38 smithi099 ceph-mon[24594]: pgmap v202: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:21:38.478 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:38 smithi002 ceph-mon[23737]: pgmap v202: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:21:38.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:38 smithi002 ceph-mon[28203]: pgmap v202: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:21:38.966 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:21:38 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: 2024-09-17T00:21:38.854+0000 7f0e25edf040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-17T00:21:39.348 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:21:38 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: 2024-09-17T00:21:38.973+0000 7f0e25edf040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-17T00:21:39.348 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:21:39 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: 2024-09-17T00:21:39.132+0000 7f0e25edf040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-17T00:21:39.348 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:21:39 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: 2024-09-17T00:21:39.200+0000 7f0e25edf040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-17T00:21:39.348 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:21:39 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: 2024-09-17T00:21:39.350+0000 7f0e25edf040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-17T00:21:39.479 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:21:39 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: ::ffff:172.21.15.99 - - [17/Sep/2024:00:21:39] "GET /metrics HTTP/1.1" 200 214851 "" "Prometheus/2.33.4" 2024-09-17T00:21:39.646 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:21:39 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: 2024-09-17T00:21:39.419+0000 7f0e25edf040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-17T00:21:39.646 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:21:39 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: 2024-09-17T00:21:39.487+0000 7f0e25edf040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-17T00:21:39.646 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:21:39 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: 2024-09-17T00:21:39.648+0000 7f0e25edf040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-17T00:21:39.935 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:21:39 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: 2024-09-17T00:21:39.715+0000 7f0e25edf040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-17T00:21:39.935 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:21:39 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: 2024-09-17T00:21:39.783+0000 7f0e25edf040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-17T00:21:39.935 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:21:39 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: 2024-09-17T00:21:39.869+0000 7f0e25edf040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-17T00:21:39.935 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:21:39 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: 2024-09-17T00:21:39.937+0000 7f0e25edf040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-17T00:21:40.216 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:40 smithi099 ceph-mon[24594]: pgmap v203: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:21:40.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:40 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:21:40.217 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:21:40 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: 2024-09-17T00:21:40.002+0000 7f0e25edf040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-17T00:21:40.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:40 smithi002 ceph-mon[28203]: pgmap v203: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:21:40.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:40 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:21:40.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:40 smithi002 ceph-mon[23737]: pgmap v203: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:21:40.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:40 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:21:41.111 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:21:40 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: 2024-09-17T00:21:40.860+0000 7f0e25edf040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-17T00:21:41.111 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:21:40 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: [17/Sep/2024:00:21:40] ENGINE Bus STARTING 2024-09-17T00:21:41.111 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:21:40 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: CherryPy Checker: 2024-09-17T00:21:41.111 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:21:40 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: The Application mounted at '' has an empty config. 2024-09-17T00:21:41.111 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:21:40 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: 2024-09-17T00:21:41.111 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:21:40 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: [17/Sep/2024:00:21:40] ENGINE Serving on http://:::9283 2024-09-17T00:21:41.111 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:21:40 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: [17/Sep/2024:00:21:40] ENGINE Bus STARTED 2024-09-17T00:21:41.872 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:41 smithi099 ceph-mon[24594]: pgmap v204: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:21:41.955 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:41 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:21:41.955 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:41 smithi099 ceph-mon[24594]: Standby manager daemon x restarted 2024-09-17T00:21:41.955 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:41 smithi099 ceph-mon[24594]: Standby manager daemon x started 2024-09-17T00:21:41.955 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:41 smithi099 ceph-mon[24594]: from='mgr.? 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-17T00:21:41.955 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:41 smithi099 ceph-mon[24594]: from='mgr.? 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-17T00:21:41.955 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:41 smithi099 ceph-mon[24594]: from='mgr.? 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-17T00:21:41.955 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:41 smithi099 ceph-mon[24594]: from='mgr.? 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-17T00:21:41.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:41 smithi002 ceph-mon[28203]: pgmap v204: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:21:41.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:41 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:21:41.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:41 smithi002 ceph-mon[28203]: Standby manager daemon x restarted 2024-09-17T00:21:41.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:41 smithi002 ceph-mon[28203]: Standby manager daemon x started 2024-09-17T00:21:41.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:41 smithi002 ceph-mon[28203]: from='mgr.? 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-17T00:21:41.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:41 smithi002 ceph-mon[28203]: from='mgr.? 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-17T00:21:41.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:41 smithi002 ceph-mon[28203]: from='mgr.? 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-17T00:21:41.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:41 smithi002 ceph-mon[28203]: from='mgr.? 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-17T00:21:41.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:41 smithi002 ceph-mon[23737]: pgmap v204: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:21:41.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:41 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:21:41.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:41 smithi002 ceph-mon[23737]: Standby manager daemon x restarted 2024-09-17T00:21:41.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:41 smithi002 ceph-mon[23737]: Standby manager daemon x started 2024-09-17T00:21:41.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:41 smithi002 ceph-mon[23737]: from='mgr.? 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-17T00:21:41.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:41 smithi002 ceph-mon[23737]: from='mgr.? 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-17T00:21:41.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:41 smithi002 ceph-mon[23737]: from='mgr.? 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-17T00:21:41.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:41 smithi002 ceph-mon[23737]: from='mgr.? 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-17T00:21:42.908 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:42 smithi002 ceph-mon[28203]: mgrmap e22: y(active, since 5m), standbys: x 2024-09-17T00:21:42.908 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:42 smithi002 ceph-mon[23737]: mgrmap e22: y(active, since 5m), standbys: x 2024-09-17T00:21:42.921 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:42 smithi099 ceph-mon[24594]: mgrmap e22: y(active, since 5m), standbys: x 2024-09-17T00:21:43.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:43 smithi099 ceph-mon[24594]: pgmap v205: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:21:43.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:43 smithi099 ceph-mon[24594]: from='client.14889 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:21:43.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:43 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:21:43.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:43 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:21:43.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:43 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T00:21:43.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:43 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-09-17T00:21:43.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:43 smithi099 ceph-mon[24594]: osdmap e97: 8 total, 8 up, 8 in 2024-09-17T00:21:43.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:43 smithi002 ceph-mon[23737]: pgmap v205: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:21:43.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:43 smithi002 ceph-mon[23737]: from='client.14889 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:21:43.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:43 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:21:43.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:43 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:21:43.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:43 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T00:21:43.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:43 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-09-17T00:21:43.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:43 smithi002 ceph-mon[23737]: osdmap e97: 8 total, 8 up, 8 in 2024-09-17T00:21:43.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:43 smithi002 ceph-mon[28203]: pgmap v205: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:21:43.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:43 smithi002 ceph-mon[28203]: from='client.14889 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:21:43.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:43 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:21:43.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:43 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' 2024-09-17T00:21:43.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:43 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T00:21:43.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:43 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-09-17T00:21:43.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:43 smithi002 ceph-mon[28203]: osdmap e97: 8 total, 8 up, 8 in 2024-09-17T00:21:43.980 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Sep 17 00:21:43 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-alertmanager-a[48044]: level=error ts=2024-09-17T00:21:43.507Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.99:8443/api/prometheus_receiver\": dial tcp 172.21.15.99:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.2:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.2 because it doesn't contain any IP SANs" 2024-09-17T00:21:43.981 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Sep 17 00:21:43 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-alertmanager-a[48044]: level=warn ts=2024-09-17T00:21:43.509Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.2:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.2 because it doesn't contain any IP SANs" 2024-09-17T00:21:43.981 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Sep 17 00:21:43 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-alertmanager-a[48044]: level=warn ts=2024-09-17T00:21:43.510Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.99:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.99 because it doesn't contain any IP SANs" 2024-09-17T00:21:44.861 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Sep 17 00:21:44 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-alertmanager-a[48044]: level=warn ts=2024-09-17T00:21:44.823Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=3 err="Post \"https://172.21.15.99:8443/api/prometheus_receiver\": dial tcp 172.21.15.99:8443: connect: connection refused" 2024-09-17T00:21:44.862 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:21:44 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: 2024-09-17T00:21:44.604+0000 7f5968dce700 -1 mgr handle_mgr_map I was active but no longer am 2024-09-17T00:21:44.862 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:21:44 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: ignoring --setuser ceph since I am not root 2024-09-17T00:21:44.863 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:21:44 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: ignoring --setgroup ceph since I am not root 2024-09-17T00:21:44.863 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:21:44 smithi002 ceph-mgr[23968]: -- 172.21.15.2:0/403262203 <== mon.0 v2:172.21.15.2:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5577f284a340 con 0x5577f2822800 2024-09-17T00:21:44.863 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:21:44 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: 2024-09-17T00:21:44.783+0000 7ff90632f000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-17T00:21:44.863 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:21:44 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: 2024-09-17T00:21:44.864+0000 7ff90632f000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-17T00:21:44.864 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:44 smithi002 ceph-mon[28203]: Upgrade: Need to upgrade myself (mgr.y) 2024-09-17T00:21:44.864 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:44 smithi002 ceph-mon[28203]: Failing over to other MGR 2024-09-17T00:21:44.864 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:44 smithi002 ceph-mon[28203]: Standby manager daemon y started 2024-09-17T00:21:44.864 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:44 smithi002 ceph-mon[28203]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd='[{"prefix": "mgr fail", "who": "y"}]': finished 2024-09-17T00:21:44.864 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:44 smithi002 ceph-mon[28203]: mgrmap e23: x(active, starting, since 0.999148s), standbys: y 2024-09-17T00:21:44.865 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:44 smithi002 ceph-mon[28203]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T00:21:44.865 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:44 smithi002 ceph-mon[28203]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T00:21:44.865 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:44 smithi002 ceph-mon[28203]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T00:21:44.865 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:44 smithi002 ceph-mon[28203]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-17T00:21:44.865 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:44 smithi002 ceph-mon[28203]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-17T00:21:44.865 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:44 smithi002 ceph-mon[28203]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T00:21:44.865 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:44 smithi002 ceph-mon[28203]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T00:21:44.865 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:44 smithi002 ceph-mon[28203]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T00:21:44.865 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:44 smithi002 ceph-mon[28203]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T00:21:44.865 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:44 smithi002 ceph-mon[28203]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T00:21:44.865 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:44 smithi002 ceph-mon[28203]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T00:21:44.866 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:44 smithi002 ceph-mon[28203]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T00:21:44.866 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:44 smithi002 ceph-mon[28203]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T00:21:44.866 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:44 smithi002 ceph-mon[28203]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-17T00:21:44.866 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:44 smithi002 ceph-mon[28203]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-17T00:21:44.866 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:44 smithi002 ceph-mon[28203]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-17T00:21:44.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:44 smithi002 ceph-mon[23737]: Upgrade: Need to upgrade myself (mgr.y) 2024-09-17T00:21:44.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:44 smithi002 ceph-mon[23737]: Failing over to other MGR 2024-09-17T00:21:44.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:44 smithi002 ceph-mon[23737]: Standby manager daemon y started 2024-09-17T00:21:44.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:44 smithi002 ceph-mon[23737]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd='[{"prefix": "mgr fail", "who": "y"}]': finished 2024-09-17T00:21:44.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:44 smithi002 ceph-mon[23737]: mgrmap e23: x(active, starting, since 0.999148s), standbys: y 2024-09-17T00:21:44.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:44 smithi002 ceph-mon[23737]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T00:21:44.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:44 smithi002 ceph-mon[23737]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T00:21:44.868 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:44 smithi002 ceph-mon[23737]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T00:21:44.868 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:44 smithi002 ceph-mon[23737]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-17T00:21:44.868 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:44 smithi002 ceph-mon[23737]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-17T00:21:44.868 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:44 smithi002 ceph-mon[23737]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T00:21:44.868 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:44 smithi002 ceph-mon[23737]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T00:21:44.868 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:44 smithi002 ceph-mon[23737]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T00:21:44.868 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:44 smithi002 ceph-mon[23737]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T00:21:44.868 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:44 smithi002 ceph-mon[23737]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T00:21:44.868 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:44 smithi002 ceph-mon[23737]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T00:21:44.868 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:44 smithi002 ceph-mon[23737]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T00:21:44.868 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:44 smithi002 ceph-mon[23737]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T00:21:44.868 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:44 smithi002 ceph-mon[23737]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-17T00:21:44.868 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:44 smithi002 ceph-mon[23737]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-17T00:21:44.868 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:44 smithi002 ceph-mon[23737]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-17T00:21:44.930 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:21:44 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: [17/Sep/2024:00:21:44] ENGINE Bus STOPPING 2024-09-17T00:21:44.930 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:44 smithi099 ceph-mon[24594]: Upgrade: Need to upgrade myself (mgr.y) 2024-09-17T00:21:44.930 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:44 smithi099 ceph-mon[24594]: Failing over to other MGR 2024-09-17T00:21:44.930 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:44 smithi099 ceph-mon[24594]: Standby manager daemon y started 2024-09-17T00:21:44.930 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:44 smithi099 ceph-mon[24594]: from='mgr.24337 172.21.15.2:0/4277289563' entity='mgr.y' cmd='[{"prefix": "mgr fail", "who": "y"}]': finished 2024-09-17T00:21:44.930 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:44 smithi099 ceph-mon[24594]: mgrmap e23: x(active, starting, since 0.999148s), standbys: y 2024-09-17T00:21:44.930 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:44 smithi099 ceph-mon[24594]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T00:21:44.930 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:44 smithi099 ceph-mon[24594]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T00:21:44.930 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:44 smithi099 ceph-mon[24594]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T00:21:44.931 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:44 smithi099 ceph-mon[24594]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-17T00:21:44.931 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:44 smithi099 ceph-mon[24594]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-17T00:21:44.931 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:44 smithi099 ceph-mon[24594]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T00:21:44.931 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:44 smithi099 ceph-mon[24594]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T00:21:44.931 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:44 smithi099 ceph-mon[24594]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T00:21:44.931 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:44 smithi099 ceph-mon[24594]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T00:21:44.931 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:44 smithi099 ceph-mon[24594]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T00:21:44.931 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:44 smithi099 ceph-mon[24594]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T00:21:44.931 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:44 smithi099 ceph-mon[24594]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T00:21:44.931 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:44 smithi099 ceph-mon[24594]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T00:21:44.931 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:44 smithi099 ceph-mon[24594]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-17T00:21:44.931 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:44 smithi099 ceph-mon[24594]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-17T00:21:44.931 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:44 smithi099 ceph-mon[24594]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-17T00:21:45.144 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:21:45 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: 2024-09-17T00:21:45.059+0000 7ff90632f000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-17T00:21:45.144 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Sep 17 00:21:45 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-alertmanager-a[48044]: level=warn ts=2024-09-17T00:21:45.049Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=3 err="Post \"https://172.21.15.2:8443//api/prometheus_receiver\": dial tcp 172.21.15.2:8443: connect: connection refused" 2024-09-17T00:21:45.194 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:21:44 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: ::ffff:172.21.15.99 - - [17/Sep/2024:00:21:44] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-09-17T00:21:45.195 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:21:44 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: [17/Sep/2024:00:21:44] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-17T00:21:45.195 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:21:44 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: [17/Sep/2024:00:21:44] ENGINE Bus STOPPED 2024-09-17T00:21:45.195 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:21:45 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: [17/Sep/2024:00:21:45] ENGINE Bus STARTING 2024-09-17T00:21:45.195 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:21:45 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: [17/Sep/2024:00:21:45] ENGINE Serving on http://:::9283 2024-09-17T00:21:45.195 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:21:45 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: [17/Sep/2024:00:21:45] ENGINE Bus STARTED 2024-09-17T00:21:45.479 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:21:45 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: 2024-09-17T00:21:45.151+0000 7ff90632f000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-17T00:21:45.479 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:21:45 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: 2024-09-17T00:21:45.289+0000 7ff90632f000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-17T00:21:45.865 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:21:45 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: 2024-09-17T00:21:45.514+0000 7ff90632f000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-17T00:21:45.866 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:45 smithi002 ceph-mon[28203]: Manager daemon x is now available 2024-09-17T00:21:45.866 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:45 smithi002 ceph-mon[28203]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:21:45.866 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:45 smithi002 ceph-mon[28203]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:21:45.866 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:45 smithi002 ceph-mon[28203]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:21:45.866 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:45 smithi002 ceph-mon[28203]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:21:45.866 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:45 smithi002 ceph-mon[28203]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T00:21:45.866 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:45 smithi002 ceph-mon[28203]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-09-17T00:21:45.866 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:45 smithi002 ceph-mon[28203]: from='mgr.24986 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-09-17T00:21:45.867 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:45 smithi002 ceph-mon[28203]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-09-17T00:21:45.867 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:45 smithi002 ceph-mon[28203]: from='mgr.24986 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-09-17T00:21:45.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:45 smithi002 ceph-mon[23737]: Manager daemon x is now available 2024-09-17T00:21:45.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:45 smithi002 ceph-mon[23737]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:21:45.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:45 smithi002 ceph-mon[23737]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:21:45.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:45 smithi002 ceph-mon[23737]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:21:45.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:45 smithi002 ceph-mon[23737]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:21:45.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:45 smithi002 ceph-mon[23737]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T00:21:45.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:45 smithi002 ceph-mon[23737]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-09-17T00:21:45.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:45 smithi002 ceph-mon[23737]: from='mgr.24986 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-09-17T00:21:45.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:45 smithi002 ceph-mon[23737]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-09-17T00:21:45.867 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:45 smithi002 ceph-mon[23737]: from='mgr.24986 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-09-17T00:21:45.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:45 smithi099 ceph-mon[24594]: Manager daemon x is now available 2024-09-17T00:21:45.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:45 smithi099 ceph-mon[24594]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:21:45.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:45 smithi099 ceph-mon[24594]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:21:45.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:45 smithi099 ceph-mon[24594]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:21:45.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:45 smithi099 ceph-mon[24594]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:21:45.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:45 smithi099 ceph-mon[24594]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T00:21:45.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:45 smithi099 ceph-mon[24594]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-09-17T00:21:45.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:45 smithi099 ceph-mon[24594]: from='mgr.24986 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-09-17T00:21:45.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:45 smithi099 ceph-mon[24594]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-09-17T00:21:45.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:45 smithi099 ceph-mon[24594]: from='mgr.24986 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-09-17T00:21:46.131 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:21:45 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: 2024-09-17T00:21:45.868+0000 7ff90632f000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-17T00:21:46.131 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:21:45 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: 2024-09-17T00:21:45.960+0000 7ff90632f000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-17T00:21:46.478 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:21:46 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: 2024-09-17T00:21:46.133+0000 7ff90632f000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-17T00:21:46.848 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Sep 17 00:21:46 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-alertmanager-a[48044]: level=warn ts=2024-09-17T00:21:46.509Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=4 err="Post \"https://172.21.15.99:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.99 because it doesn't contain any IP SANs" 2024-09-17T00:21:46.849 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:46 smithi002 ceph-mon[28203]: Queued rgw.foo for migration 2024-09-17T00:21:46.849 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:46 smithi002 ceph-mon[28203]: Queued rgw.smpl for migration 2024-09-17T00:21:46.849 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:46 smithi002 ceph-mon[28203]: No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'foo', 'service_name': 'rgw.foo', 'service_type': 'rgw', 'spec': {'rgw_frontend_port': 8000, 'rgw_realm': 'r', 'rgw_zone': 'z'}} 2024-09-17T00:21:46.849 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:46 smithi002 ceph-mon[28203]: No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'smpl', 'service_name': 'rgw.smpl', 'service_type': 'rgw'} 2024-09-17T00:21:46.849 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:46 smithi002 ceph-mon[28203]: Migrating certs/keys for iscsi.foo spec to cert store 2024-09-17T00:21:46.849 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:46 smithi002 ceph-mon[28203]: Migrating certs/keys for rgw.foo spec to cert store 2024-09-17T00:21:46.849 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:46 smithi002 ceph-mon[28203]: Migrating certs/keys for rgw.smpl spec to cert store 2024-09-17T00:21:46.849 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:46 smithi002 ceph-mon[28203]: Checking for cert/key for grafana.a 2024-09-17T00:21:46.849 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:46 smithi002 ceph-mon[28203]: Deploying cephadm binary to smithi002 2024-09-17T00:21:46.849 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:46 smithi002 ceph-mon[28203]: Deploying cephadm binary to smithi099 2024-09-17T00:21:46.849 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:46 smithi002 ceph-mon[28203]: mgrmap e24: x(active, since 2s), standbys: y 2024-09-17T00:21:46.850 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:46 smithi002 ceph-mon[23737]: Queued rgw.foo for migration 2024-09-17T00:21:46.850 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:46 smithi002 ceph-mon[23737]: Queued rgw.smpl for migration 2024-09-17T00:21:46.850 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:46 smithi002 ceph-mon[23737]: No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'foo', 'service_name': 'rgw.foo', 'service_type': 'rgw', 'spec': {'rgw_frontend_port': 8000, 'rgw_realm': 'r', 'rgw_zone': 'z'}} 2024-09-17T00:21:46.850 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:46 smithi002 ceph-mon[23737]: No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'smpl', 'service_name': 'rgw.smpl', 'service_type': 'rgw'} 2024-09-17T00:21:46.850 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:46 smithi002 ceph-mon[23737]: Migrating certs/keys for iscsi.foo spec to cert store 2024-09-17T00:21:46.850 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:46 smithi002 ceph-mon[23737]: Migrating certs/keys for rgw.foo spec to cert store 2024-09-17T00:21:46.850 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:46 smithi002 ceph-mon[23737]: Migrating certs/keys for rgw.smpl spec to cert store 2024-09-17T00:21:46.850 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:46 smithi002 ceph-mon[23737]: Checking for cert/key for grafana.a 2024-09-17T00:21:46.850 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:46 smithi002 ceph-mon[23737]: Deploying cephadm binary to smithi002 2024-09-17T00:21:46.850 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:46 smithi002 ceph-mon[23737]: Deploying cephadm binary to smithi099 2024-09-17T00:21:46.851 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:46 smithi002 ceph-mon[23737]: mgrmap e24: x(active, since 2s), standbys: y 2024-09-17T00:21:46.966 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:46 smithi099 ceph-mon[24594]: Queued rgw.foo for migration 2024-09-17T00:21:46.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:46 smithi099 ceph-mon[24594]: Queued rgw.smpl for migration 2024-09-17T00:21:46.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:46 smithi099 ceph-mon[24594]: No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'foo', 'service_name': 'rgw.foo', 'service_type': 'rgw', 'spec': {'rgw_frontend_port': 8000, 'rgw_realm': 'r', 'rgw_zone': 'z'}} 2024-09-17T00:21:46.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:46 smithi099 ceph-mon[24594]: No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'smpl', 'service_name': 'rgw.smpl', 'service_type': 'rgw'} 2024-09-17T00:21:46.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:46 smithi099 ceph-mon[24594]: Migrating certs/keys for iscsi.foo spec to cert store 2024-09-17T00:21:46.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:46 smithi099 ceph-mon[24594]: Migrating certs/keys for rgw.foo spec to cert store 2024-09-17T00:21:46.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:46 smithi099 ceph-mon[24594]: Migrating certs/keys for rgw.smpl spec to cert store 2024-09-17T00:21:46.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:46 smithi099 ceph-mon[24594]: Checking for cert/key for grafana.a 2024-09-17T00:21:46.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:46 smithi099 ceph-mon[24594]: Deploying cephadm binary to smithi002 2024-09-17T00:21:46.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:46 smithi099 ceph-mon[24594]: Deploying cephadm binary to smithi099 2024-09-17T00:21:46.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:46 smithi099 ceph-mon[24594]: mgrmap e24: x(active, since 2s), standbys: y 2024-09-17T00:21:47.586 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:21:47 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: 2024-09-17T00:21:47.271+0000 7ff90632f000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-17T00:21:47.586 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:21:47 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: 2024-09-17T00:21:47.386+0000 7ff90632f000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-17T00:21:47.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:47 smithi099 ceph-mon[24594]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:21:47.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:47 smithi099 ceph-mon[24594]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:21:47.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:47 smithi099 ceph-mon[24594]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:21:47.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:47 smithi099 ceph-mon[24594]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:21:47.967 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:21:47 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: 2024-09-17T00:21:47.656+0000 7f0d78e83640 -1 log_channel(cephadm) log [ERR] : [17/Sep/2024:00:21:47] ENGINE Error in HTTPServer.serve 2024-09-17T00:21:47.967 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:21:47 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: Traceback (most recent call last): 2024-09-17T00:21:47.967 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:21:47 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-17T00:21:47.968 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:21:47 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: self._connections.run(self.expiration_interval) 2024-09-17T00:21:47.968 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:21:47 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-17T00:21:47.968 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:21:47 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: self._run(expiration_interval) 2024-09-17T00:21:47.968 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:21:47 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-17T00:21:47.968 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:21:47 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: new_conn = self._from_server_socket(self.server.socket) 2024-09-17T00:21:47.968 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:21:47 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-17T00:21:47.968 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:21:47 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-17T00:21:47.968 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:21:47 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-17T00:21:47.968 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:21:47 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: s = self.context.wrap_socket( 2024-09-17T00:21:47.968 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:21:47 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-17T00:21:47.968 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:21:47 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: return self.sslsocket_class._create( 2024-09-17T00:21:47.968 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:21:47 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-17T00:21:47.968 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:21:47 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: self.do_handshake() 2024-09-17T00:21:47.969 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:21:47 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-17T00:21:47.969 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:21:47 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: self._sslobj.do_handshake() 2024-09-17T00:21:47.969 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:21:47 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-17T00:21:47.969 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:21:47 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: 2024-09-17T00:21:47.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:47 smithi002 ceph-mon[28203]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:21:47.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:47 smithi002 ceph-mon[28203]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:21:47.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:47 smithi002 ceph-mon[28203]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:21:47.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:47 smithi002 ceph-mon[28203]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:21:47.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:47 smithi002 ceph-mon[23737]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:21:47.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:47 smithi002 ceph-mon[23737]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:21:47.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:47 smithi002 ceph-mon[23737]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:21:47.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:47 smithi002 ceph-mon[23737]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:21:48.684 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:21:48 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: 2024-09-17T00:21:48.396+0000 7ff90632f000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-17T00:21:48.960 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:21:48 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: 2024-09-17T00:21:48.724+0000 7ff90632f000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-17T00:21:48.960 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:21:48 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: 2024-09-17T00:21:48.867+0000 7ff90632f000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-17T00:21:48.960 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:48 smithi002 ceph-mon[23737]: [17/Sep/2024:00:21:47] ENGINE Bus STARTING 2024-09-17T00:21:48.960 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:48 smithi002 ceph-mon[23737]: [17/Sep/2024:00:21:47] ENGINE Serving on http://172.21.15.99:8765 2024-09-17T00:21:48.960 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:48 smithi002 ceph-mon[23737]: [17/Sep/2024:00:21:47] ENGINE Serving on https://172.21.15.99:7150 2024-09-17T00:21:48.960 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:48 smithi002 ceph-mon[23737]: [17/Sep/2024:00:21:47] ENGINE Bus STARTED 2024-09-17T00:21:48.961 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:48 smithi002 ceph-mon[23737]: [17/Sep/2024:00:21:47] ENGINE Error in HTTPServer.serve 2024-09-17T00:21:48.961 INFO:journalctl@ceph.mon.a.smithi002.stdout: Traceback (most recent call last): 2024-09-17T00:21:48.961 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-17T00:21:48.961 INFO:journalctl@ceph.mon.a.smithi002.stdout: self._connections.run(self.expiration_interval) 2024-09-17T00:21:48.961 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-17T00:21:48.961 INFO:journalctl@ceph.mon.a.smithi002.stdout: self._run(expiration_interval) 2024-09-17T00:21:48.961 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-17T00:21:48.961 INFO:journalctl@ceph.mon.a.smithi002.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-17T00:21:48.961 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-17T00:21:48.961 INFO:journalctl@ceph.mon.a.smithi002.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-17T00:21:48.961 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-17T00:21:48.961 INFO:journalctl@ceph.mon.a.smithi002.stdout: s = self.context.wrap_socket( 2024-09-17T00:21:48.961 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-17T00:21:48.961 INFO:journalctl@ceph.mon.a.smithi002.stdout: return self.sslsocket_class._create( 2024-09-17T00:21:48.962 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-17T00:21:48.962 INFO:journalctl@ceph.mon.a.smithi002.stdout: self.do_handshake() 2024-09-17T00:21:48.962 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-17T00:21:48.962 INFO:journalctl@ceph.mon.a.smithi002.stdout: self._sslobj.do_handshake() 2024-09-17T00:21:48.962 INFO:journalctl@ceph.mon.a.smithi002.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-17T00:21:48.962 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:48 smithi002 ceph-mon[23737]: mgrmap e25: x(active, since 4s), standbys: y 2024-09-17T00:21:48.962 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:48 smithi002 ceph-mon[28203]: [17/Sep/2024:00:21:47] ENGINE Bus STARTING 2024-09-17T00:21:48.962 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:48 smithi002 ceph-mon[28203]: [17/Sep/2024:00:21:47] ENGINE Serving on http://172.21.15.99:8765 2024-09-17T00:21:48.962 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:48 smithi002 ceph-mon[28203]: [17/Sep/2024:00:21:47] ENGINE Serving on https://172.21.15.99:7150 2024-09-17T00:21:48.962 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:48 smithi002 ceph-mon[28203]: [17/Sep/2024:00:21:47] ENGINE Bus STARTED 2024-09-17T00:21:48.962 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:48 smithi002 ceph-mon[28203]: [17/Sep/2024:00:21:47] ENGINE Error in HTTPServer.serve 2024-09-17T00:21:48.963 INFO:journalctl@ceph.mon.c.smithi002.stdout: Traceback (most recent call last): 2024-09-17T00:21:48.963 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-17T00:21:48.963 INFO:journalctl@ceph.mon.c.smithi002.stdout: self._connections.run(self.expiration_interval) 2024-09-17T00:21:48.963 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-17T00:21:48.963 INFO:journalctl@ceph.mon.c.smithi002.stdout: self._run(expiration_interval) 2024-09-17T00:21:48.963 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-17T00:21:48.963 INFO:journalctl@ceph.mon.c.smithi002.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-17T00:21:48.963 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-17T00:21:48.963 INFO:journalctl@ceph.mon.c.smithi002.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-17T00:21:48.963 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-17T00:21:48.963 INFO:journalctl@ceph.mon.c.smithi002.stdout: s = self.context.wrap_socket( 2024-09-17T00:21:48.963 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-17T00:21:48.963 INFO:journalctl@ceph.mon.c.smithi002.stdout: return self.sslsocket_class._create( 2024-09-17T00:21:48.963 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-17T00:21:48.964 INFO:journalctl@ceph.mon.c.smithi002.stdout: self.do_handshake() 2024-09-17T00:21:48.964 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-17T00:21:48.964 INFO:journalctl@ceph.mon.c.smithi002.stdout: self._sslobj.do_handshake() 2024-09-17T00:21:48.964 INFO:journalctl@ceph.mon.c.smithi002.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-17T00:21:48.964 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:48 smithi002 ceph-mon[28203]: mgrmap e25: x(active, since 4s), standbys: y 2024-09-17T00:21:48.966 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:48 smithi099 ceph-mon[24594]: [17/Sep/2024:00:21:47] ENGINE Bus STARTING 2024-09-17T00:21:48.966 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:48 smithi099 ceph-mon[24594]: [17/Sep/2024:00:21:47] ENGINE Serving on http://172.21.15.99:8765 2024-09-17T00:21:48.966 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:48 smithi099 ceph-mon[24594]: [17/Sep/2024:00:21:47] ENGINE Serving on https://172.21.15.99:7150 2024-09-17T00:21:48.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:48 smithi099 ceph-mon[24594]: [17/Sep/2024:00:21:47] ENGINE Bus STARTED 2024-09-17T00:21:48.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:48 smithi099 ceph-mon[24594]: [17/Sep/2024:00:21:47] ENGINE Error in HTTPServer.serve 2024-09-17T00:21:48.967 INFO:journalctl@ceph.mon.b.smithi099.stdout: Traceback (most recent call last): 2024-09-17T00:21:48.967 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-17T00:21:48.967 INFO:journalctl@ceph.mon.b.smithi099.stdout: self._connections.run(self.expiration_interval) 2024-09-17T00:21:48.967 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-17T00:21:48.967 INFO:journalctl@ceph.mon.b.smithi099.stdout: self._run(expiration_interval) 2024-09-17T00:21:48.967 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-17T00:21:48.967 INFO:journalctl@ceph.mon.b.smithi099.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-17T00:21:48.967 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-17T00:21:48.967 INFO:journalctl@ceph.mon.b.smithi099.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-17T00:21:48.967 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-17T00:21:48.967 INFO:journalctl@ceph.mon.b.smithi099.stdout: s = self.context.wrap_socket( 2024-09-17T00:21:48.968 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-17T00:21:48.968 INFO:journalctl@ceph.mon.b.smithi099.stdout: return self.sslsocket_class._create( 2024-09-17T00:21:48.968 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-17T00:21:48.968 INFO:journalctl@ceph.mon.b.smithi099.stdout: self.do_handshake() 2024-09-17T00:21:48.968 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-17T00:21:48.968 INFO:journalctl@ceph.mon.b.smithi099.stdout: self._sslobj.do_handshake() 2024-09-17T00:21:48.968 INFO:journalctl@ceph.mon.b.smithi099.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-17T00:21:48.968 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:48 smithi099 ceph-mon[24594]: mgrmap e25: x(active, since 4s), standbys: y 2024-09-17T00:21:49.229 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:21:48 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: 2024-09-17T00:21:48.961+0000 7ff90632f000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-17T00:21:49.229 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:21:49 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: 2024-09-17T00:21:49.075+0000 7ff90632f000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-17T00:21:49.229 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:21:49 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: 2024-09-17T00:21:49.171+0000 7ff90632f000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-17T00:21:49.528 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:21:49 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: 2024-09-17T00:21:49.333+0000 7ff90632f000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-17T00:21:49.529 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:21:49 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: 2024-09-17T00:21:49.433+0000 7ff90632f000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-17T00:21:49.854 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:49 smithi002 ceph-mon[23737]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:21:49.854 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:49 smithi002 ceph-mon[28203]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:21:49.855 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:21:49 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: 2024-09-17T00:21:49.537+0000 7ff90632f000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-17T00:21:49.855 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:21:49 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: 2024-09-17T00:21:49.856+0000 7ff90632f000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-17T00:21:49.966 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:49 smithi099 ceph-mon[24594]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:21:50.189 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:21:49 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: 2024-09-17T00:21:49.955+0000 7ff90632f000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-17T00:21:50.189 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:21:50 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: 2024-09-17T00:21:50.050+0000 7ff90632f000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-17T00:21:50.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:50 smithi002 ceph-mon[28203]: mgrmap e26: x(active, since 6s), standbys: y 2024-09-17T00:21:50.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:50 smithi002 ceph-mon[28203]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:21:50.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:50 smithi002 ceph-mon[23737]: mgrmap e26: x(active, since 6s), standbys: y 2024-09-17T00:21:50.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:50 smithi002 ceph-mon[23737]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:21:51.216 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:50 smithi099 ceph-mon[24594]: mgrmap e26: x(active, since 6s), standbys: y 2024-09-17T00:21:51.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:50 smithi099 ceph-mon[24594]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:21:51.479 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:21:51 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: 2024-09-17T00:21:51.210+0000 7ff90632f000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-17T00:21:51.479 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:21:51 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: [17/Sep/2024:00:21:51] ENGINE Bus STARTING 2024-09-17T00:21:51.479 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:21:51 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: CherryPy Checker: 2024-09-17T00:21:51.480 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:21:51 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: The Application mounted at '' has an empty config. 2024-09-17T00:21:51.480 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:21:51 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: 2024-09-17T00:21:51.480 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:21:51 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: [17/Sep/2024:00:21:51] ENGINE Serving on http://:::9283 2024-09-17T00:21:51.480 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:21:51 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: [17/Sep/2024:00:21:51] ENGINE Bus STARTED 2024-09-17T00:21:52.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:51 smithi099 ceph-mon[24594]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:21:52.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:51 smithi099 ceph-mon[24594]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:21:52.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:51 smithi099 ceph-mon[24594]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:21:52.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:51 smithi099 ceph-mon[24594]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:21:52.218 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:51 smithi099 ceph-mon[24594]: Standby manager daemon y restarted 2024-09-17T00:21:52.218 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:51 smithi099 ceph-mon[24594]: Standby manager daemon y started 2024-09-17T00:21:52.218 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:51 smithi099 ceph-mon[24594]: from='mgr.? 172.21.15.2:0/3540023627' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-17T00:21:52.218 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:51 smithi099 ceph-mon[24594]: from='mgr.? 172.21.15.2:0/3540023627' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-17T00:21:52.218 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:51 smithi099 ceph-mon[24594]: from='mgr.? 172.21.15.2:0/3540023627' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-09-17T00:21:52.218 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:51 smithi099 ceph-mon[24594]: from='mgr.? 172.21.15.2:0/3540023627' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-17T00:21:52.218 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:51 smithi099 ceph-mon[24594]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:21:52.218 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:51 smithi099 ceph-mon[24594]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:21:52.218 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:51 smithi099 ceph-mon[24594]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:21:52.218 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:51 smithi099 ceph-mon[24594]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:21:52.218 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:51 smithi099 ceph-mon[24594]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-17T00:21:52.218 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:51 smithi099 ceph-mon[24594]: from='mgr.24986 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-17T00:21:52.218 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:51 smithi099 ceph-mon[24594]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-17T00:21:52.219 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:51 smithi099 ceph-mon[24594]: from='mgr.24986 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-17T00:21:52.219 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:51 smithi099 ceph-mon[24594]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-17T00:21:52.219 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:51 smithi099 ceph-mon[24594]: from='mgr.24986 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-17T00:21:52.219 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:51 smithi099 ceph-mon[24594]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-17T00:21:52.219 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:51 smithi099 ceph-mon[24594]: from='mgr.24986 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-17T00:21:52.219 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:51 smithi099 ceph-mon[24594]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:21:52.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:51 smithi002 ceph-mon[23737]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:21:52.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:51 smithi002 ceph-mon[23737]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:21:52.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:51 smithi002 ceph-mon[23737]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:21:52.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:51 smithi002 ceph-mon[23737]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:21:52.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:51 smithi002 ceph-mon[23737]: Standby manager daemon y restarted 2024-09-17T00:21:52.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:51 smithi002 ceph-mon[23737]: Standby manager daemon y started 2024-09-17T00:21:52.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:51 smithi002 ceph-mon[23737]: from='mgr.? 172.21.15.2:0/3540023627' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-17T00:21:52.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:51 smithi002 ceph-mon[23737]: from='mgr.? 172.21.15.2:0/3540023627' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-17T00:21:52.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:51 smithi002 ceph-mon[23737]: from='mgr.? 172.21.15.2:0/3540023627' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-09-17T00:21:52.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:51 smithi002 ceph-mon[23737]: from='mgr.? 172.21.15.2:0/3540023627' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-17T00:21:52.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:51 smithi002 ceph-mon[23737]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:21:52.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:51 smithi002 ceph-mon[23737]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:21:52.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:51 smithi002 ceph-mon[23737]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:21:52.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:51 smithi002 ceph-mon[23737]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:21:52.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:51 smithi002 ceph-mon[23737]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-17T00:21:52.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:51 smithi002 ceph-mon[23737]: from='mgr.24986 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-17T00:21:52.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:51 smithi002 ceph-mon[23737]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-17T00:21:52.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:51 smithi002 ceph-mon[23737]: from='mgr.24986 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-17T00:21:52.231 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:51 smithi002 ceph-mon[23737]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-17T00:21:52.231 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:51 smithi002 ceph-mon[23737]: from='mgr.24986 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-17T00:21:52.231 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:51 smithi002 ceph-mon[23737]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-17T00:21:52.231 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:51 smithi002 ceph-mon[23737]: from='mgr.24986 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-17T00:21:52.231 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:51 smithi002 ceph-mon[23737]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:21:52.231 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:51 smithi002 ceph-mon[28203]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:21:52.231 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:51 smithi002 ceph-mon[28203]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:21:52.231 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:51 smithi002 ceph-mon[28203]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:21:52.231 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:51 smithi002 ceph-mon[28203]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:21:52.231 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:51 smithi002 ceph-mon[28203]: Standby manager daemon y restarted 2024-09-17T00:21:52.232 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:51 smithi002 ceph-mon[28203]: Standby manager daemon y started 2024-09-17T00:21:52.232 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:51 smithi002 ceph-mon[28203]: from='mgr.? 172.21.15.2:0/3540023627' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-17T00:21:52.232 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:51 smithi002 ceph-mon[28203]: from='mgr.? 172.21.15.2:0/3540023627' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-17T00:21:52.232 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:51 smithi002 ceph-mon[28203]: from='mgr.? 172.21.15.2:0/3540023627' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-09-17T00:21:52.232 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:51 smithi002 ceph-mon[28203]: from='mgr.? 172.21.15.2:0/3540023627' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-17T00:21:52.232 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:51 smithi002 ceph-mon[28203]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:21:52.232 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:51 smithi002 ceph-mon[28203]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:21:52.232 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:51 smithi002 ceph-mon[28203]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:21:52.232 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:51 smithi002 ceph-mon[28203]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:21:52.232 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:51 smithi002 ceph-mon[28203]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-17T00:21:52.232 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:51 smithi002 ceph-mon[28203]: from='mgr.24986 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-17T00:21:52.232 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:51 smithi002 ceph-mon[28203]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-17T00:21:52.232 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:51 smithi002 ceph-mon[28203]: from='mgr.24986 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-17T00:21:52.232 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:51 smithi002 ceph-mon[28203]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-17T00:21:52.233 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:51 smithi002 ceph-mon[28203]: from='mgr.24986 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-17T00:21:52.233 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:51 smithi002 ceph-mon[28203]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-17T00:21:52.233 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:51 smithi002 ceph-mon[28203]: from='mgr.24986 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-17T00:21:52.233 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:51 smithi002 ceph-mon[28203]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:21:53.400 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:53 smithi099 ceph-mon[24594]: Adjusting osd_memory_target on smithi002 to 2737M 2024-09-17T00:21:53.400 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:53 smithi099 ceph-mon[24594]: mgrmap e27: x(active, since 8s), standbys: y 2024-09-17T00:21:53.400 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:53 smithi099 ceph-mon[24594]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 31 KiB/s rd, 0 B/s wr, 17 op/s 2024-09-17T00:21:53.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:53 smithi002 ceph-mon[23737]: Adjusting osd_memory_target on smithi002 to 2737M 2024-09-17T00:21:53.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:53 smithi002 ceph-mon[23737]: mgrmap e27: x(active, since 8s), standbys: y 2024-09-17T00:21:53.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:53 smithi002 ceph-mon[23737]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 31 KiB/s rd, 0 B/s wr, 17 op/s 2024-09-17T00:21:53.480 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:53 smithi002 ceph-mon[28203]: Adjusting osd_memory_target on smithi002 to 2737M 2024-09-17T00:21:53.480 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:53 smithi002 ceph-mon[28203]: mgrmap e27: x(active, since 8s), standbys: y 2024-09-17T00:21:53.480 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:53 smithi002 ceph-mon[28203]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 31 KiB/s rd, 0 B/s wr, 17 op/s 2024-09-17T00:21:53.979 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Sep 17 00:21:53 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-alertmanager-a[48044]: level=error ts=2024-09-17T00:21:53.508Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.2:8443//api/prometheus_receiver\": dial tcp 172.21.15.2:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.99:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.99 because it doesn't contain any IP SANs" 2024-09-17T00:21:53.979 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Sep 17 00:21:53 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-alertmanager-a[48044]: level=warn ts=2024-09-17T00:21:53.511Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.2:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.2 because it doesn't contain any IP SANs" 2024-09-17T00:21:53.979 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Sep 17 00:21:53 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-alertmanager-a[48044]: level=warn ts=2024-09-17T00:21:53.511Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.99:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.99 because it doesn't contain any IP SANs" 2024-09-17T00:21:54.335 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:54 smithi099 ceph-mon[24594]: from='client.14889 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:21:54.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:54 smithi002 ceph-mon[28203]: from='client.14889 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:21:54.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:54 smithi002 ceph-mon[23737]: from='client.14889 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:21:55.216 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:21:54 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: ::ffff:172.21.15.99 - - [17/Sep/2024:00:21:54] "GET /metrics HTTP/1.1" 200 35334 "" "Prometheus/2.33.4" 2024-09-17T00:21:55.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:55 smithi099 ceph-mon[24594]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 0 B/s wr, 13 op/s 2024-09-17T00:21:55.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:55 smithi099 ceph-mon[24594]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:21:55.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:55 smithi099 ceph-mon[24594]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:21:55.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:55 smithi099 ceph-mon[24594]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-17T00:21:55.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:55 smithi099 ceph-mon[24594]: from='mgr.24986 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-17T00:21:55.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:55 smithi099 ceph-mon[24594]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-17T00:21:55.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:55 smithi099 ceph-mon[24594]: from='mgr.24986 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-17T00:21:55.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:55 smithi099 ceph-mon[24594]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-17T00:21:55.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:55 smithi099 ceph-mon[24594]: from='mgr.24986 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-17T00:21:55.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:55 smithi099 ceph-mon[24594]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-17T00:21:55.968 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:55 smithi099 ceph-mon[24594]: from='mgr.24986 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-17T00:21:55.968 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:55 smithi099 ceph-mon[24594]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:21:55.968 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:55 smithi099 ceph-mon[24594]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:21:55.968 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:55 smithi099 ceph-mon[24594]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:21:55.968 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:55 smithi099 ceph-mon[24594]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:21:55.968 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:55 smithi099 ceph-mon[24594]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:21:55.968 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:55 smithi099 ceph-mon[24594]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:21:55.968 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:55 smithi099 ceph-mon[24594]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:21:55.968 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:55 smithi099 ceph-mon[24594]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:21:55.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:55 smithi002 ceph-mon[28203]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 0 B/s wr, 13 op/s 2024-09-17T00:21:55.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:55 smithi002 ceph-mon[28203]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:21:55.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:55 smithi002 ceph-mon[28203]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:21:55.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:55 smithi002 ceph-mon[28203]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-17T00:21:55.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:55 smithi002 ceph-mon[28203]: from='mgr.24986 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-17T00:21:55.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:55 smithi002 ceph-mon[28203]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-17T00:21:55.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:55 smithi002 ceph-mon[28203]: from='mgr.24986 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-17T00:21:55.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:55 smithi002 ceph-mon[28203]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-17T00:21:55.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:55 smithi002 ceph-mon[28203]: from='mgr.24986 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-17T00:21:55.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:55 smithi002 ceph-mon[28203]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-17T00:21:55.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:55 smithi002 ceph-mon[28203]: from='mgr.24986 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-17T00:21:55.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:55 smithi002 ceph-mon[28203]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:21:55.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:55 smithi002 ceph-mon[28203]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:21:55.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:55 smithi002 ceph-mon[28203]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:21:55.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:55 smithi002 ceph-mon[28203]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:21:55.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:55 smithi002 ceph-mon[28203]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:21:55.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:55 smithi002 ceph-mon[28203]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:21:55.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:55 smithi002 ceph-mon[28203]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:21:55.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:55 smithi002 ceph-mon[28203]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:21:55.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:55 smithi002 ceph-mon[23737]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 0 B/s wr, 13 op/s 2024-09-17T00:21:55.981 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:55 smithi002 ceph-mon[23737]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:21:55.981 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:55 smithi002 ceph-mon[23737]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:21:55.981 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:55 smithi002 ceph-mon[23737]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-17T00:21:55.981 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:55 smithi002 ceph-mon[23737]: from='mgr.24986 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-17T00:21:55.981 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:55 smithi002 ceph-mon[23737]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-17T00:21:55.981 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:55 smithi002 ceph-mon[23737]: from='mgr.24986 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-17T00:21:55.981 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:55 smithi002 ceph-mon[23737]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-17T00:21:55.981 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:55 smithi002 ceph-mon[23737]: from='mgr.24986 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-17T00:21:55.981 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:55 smithi002 ceph-mon[23737]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-17T00:21:55.982 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:55 smithi002 ceph-mon[23737]: from='mgr.24986 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-17T00:21:55.982 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:55 smithi002 ceph-mon[23737]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:21:55.982 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:55 smithi002 ceph-mon[23737]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:21:55.982 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:55 smithi002 ceph-mon[23737]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:21:55.982 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:55 smithi002 ceph-mon[23737]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:21:55.982 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:55 smithi002 ceph-mon[23737]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:21:55.982 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:55 smithi002 ceph-mon[23737]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:21:55.982 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:55 smithi002 ceph-mon[23737]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:21:55.982 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:55 smithi002 ceph-mon[23737]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:21:56.479 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Sep 17 00:21:56 smithi002 systemd[1]: Stopping Ceph node-exporter.a for 3aad27a2-7489-11ef-bceb-c7b262605968... 2024-09-17T00:21:56.832 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Sep 17 00:21:56 smithi002 podman[57120]: 2024-09-17 00:21:56.544571342 +0000 UTC m=+0.114131890 container died d28913b4af3091e941ac8b2d21e27d43655ae89afc90ae230f8a963cb0d31a11 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-17T00:21:56.833 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:56 smithi002 ceph-mon[23737]: Adjusting osd_memory_target on smithi099 to 2992M 2024-09-17T00:21:56.833 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:56 smithi002 ceph-mon[23737]: Updating smithi002:/etc/ceph/ceph.conf 2024-09-17T00:21:56.833 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:56 smithi002 ceph-mon[23737]: Updating smithi099:/etc/ceph/ceph.conf 2024-09-17T00:21:56.833 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:56 smithi002 ceph-mon[23737]: Updating smithi099:/var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-17T00:21:56.833 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:56 smithi002 ceph-mon[23737]: Updating smithi002:/var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-17T00:21:56.833 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:56 smithi002 ceph-mon[23737]: Updating smithi099:/etc/ceph/ceph.client.admin.keyring 2024-09-17T00:21:56.833 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:56 smithi002 ceph-mon[23737]: Updating smithi002:/etc/ceph/ceph.client.admin.keyring 2024-09-17T00:21:56.833 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:56 smithi002 ceph-mon[23737]: Updating smithi099:/var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-17T00:21:56.833 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:56 smithi002 ceph-mon[23737]: Updating smithi002:/var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-17T00:21:56.834 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:56 smithi002 ceph-mon[23737]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:21:56.834 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:56 smithi002 ceph-mon[23737]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:21:56.834 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:56 smithi002 ceph-mon[23737]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:21:56.834 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:56 smithi002 ceph-mon[23737]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:21:56.834 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:56 smithi002 ceph-mon[28203]: Adjusting osd_memory_target on smithi099 to 2992M 2024-09-17T00:21:56.834 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:56 smithi002 ceph-mon[28203]: Updating smithi002:/etc/ceph/ceph.conf 2024-09-17T00:21:56.834 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:56 smithi002 ceph-mon[28203]: Updating smithi099:/etc/ceph/ceph.conf 2024-09-17T00:21:56.834 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:56 smithi002 ceph-mon[28203]: Updating smithi099:/var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-17T00:21:56.834 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:56 smithi002 ceph-mon[28203]: Updating smithi002:/var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-17T00:21:56.835 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:56 smithi002 ceph-mon[28203]: Updating smithi099:/etc/ceph/ceph.client.admin.keyring 2024-09-17T00:21:56.835 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:56 smithi002 ceph-mon[28203]: Updating smithi002:/etc/ceph/ceph.client.admin.keyring 2024-09-17T00:21:56.835 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:56 smithi002 ceph-mon[28203]: Updating smithi099:/var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-17T00:21:56.835 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:56 smithi002 ceph-mon[28203]: Updating smithi002:/var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-17T00:21:56.835 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:56 smithi002 ceph-mon[28203]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:21:56.835 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:56 smithi002 ceph-mon[28203]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:21:56.835 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:56 smithi002 ceph-mon[28203]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:21:56.835 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:56 smithi002 ceph-mon[28203]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:21:56.963 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:56 smithi099 ceph-mon[24594]: Adjusting osd_memory_target on smithi099 to 2992M 2024-09-17T00:21:56.963 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:56 smithi099 ceph-mon[24594]: Updating smithi002:/etc/ceph/ceph.conf 2024-09-17T00:21:56.963 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:56 smithi099 ceph-mon[24594]: Updating smithi099:/etc/ceph/ceph.conf 2024-09-17T00:21:56.963 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:56 smithi099 ceph-mon[24594]: Updating smithi099:/var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-17T00:21:56.963 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:56 smithi099 ceph-mon[24594]: Updating smithi002:/var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-17T00:21:56.963 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:56 smithi099 ceph-mon[24594]: Updating smithi099:/etc/ceph/ceph.client.admin.keyring 2024-09-17T00:21:56.963 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:56 smithi099 ceph-mon[24594]: Updating smithi002:/etc/ceph/ceph.client.admin.keyring 2024-09-17T00:21:56.963 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:56 smithi099 ceph-mon[24594]: Updating smithi099:/var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-17T00:21:56.963 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:56 smithi099 ceph-mon[24594]: Updating smithi002:/var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-17T00:21:56.964 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:56 smithi099 ceph-mon[24594]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:21:56.964 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:56 smithi099 ceph-mon[24594]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:21:56.964 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:56 smithi099 ceph-mon[24594]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:21:56.964 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:56 smithi099 ceph-mon[24594]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:21:57.229 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Sep 17 00:21:57 smithi002 podman[57120]: 2024-09-17 00:21:57.051929526 +0000 UTC m=+0.621490070 container cleanup d28913b4af3091e941ac8b2d21e27d43655ae89afc90ae230f8a963cb0d31a11 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-17T00:21:57.229 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Sep 17 00:21:57 smithi002 bash[57120]: ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-a 2024-09-17T00:21:57.616 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Sep 17 00:21:57 smithi002 podman[57133]: 2024-09-17 00:21:57.353813596 +0000 UTC m=+0.809043609 container remove d28913b4af3091e941ac8b2d21e27d43655ae89afc90ae230f8a963cb0d31a11 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-17T00:21:57.616 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Sep 17 00:21:57 smithi002 systemd[1]: ceph-3aad27a2-7489-11ef-bceb-c7b262605968@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-09-17T00:21:57.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:57 smithi099 ceph-mon[24594]: Reconfiguring node-exporter.a (dependencies changed)... 2024-09-17T00:21:57.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:57 smithi099 ceph-mon[24594]: Deploying daemon node-exporter.a on smithi002 2024-09-17T00:21:57.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:57 smithi099 ceph-mon[24594]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-17T00:21:57.981 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:57 smithi002 ceph-mon[28203]: Reconfiguring node-exporter.a (dependencies changed)... 2024-09-17T00:21:57.981 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:57 smithi002 ceph-mon[28203]: Deploying daemon node-exporter.a on smithi002 2024-09-17T00:21:57.981 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:57 smithi002 ceph-mon[28203]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-17T00:21:57.981 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:57 smithi002 ceph-mon[23737]: Reconfiguring node-exporter.a (dependencies changed)... 2024-09-17T00:21:57.981 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:57 smithi002 ceph-mon[23737]: Deploying daemon node-exporter.a on smithi002 2024-09-17T00:21:57.981 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:57 smithi002 ceph-mon[23737]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-17T00:21:57.981 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Sep 17 00:21:57 smithi002 systemd[1]: ceph-3aad27a2-7489-11ef-bceb-c7b262605968@node-exporter.a.service: Failed with result 'exit-code'. 2024-09-17T00:21:57.982 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Sep 17 00:21:57 smithi002 systemd[1]: Stopped Ceph node-exporter.a for 3aad27a2-7489-11ef-bceb-c7b262605968. 2024-09-17T00:21:57.982 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Sep 17 00:21:57 smithi002 systemd[1]: ceph-3aad27a2-7489-11ef-bceb-c7b262605968@node-exporter.a.service: Consumed 3.228s CPU time. 2024-09-17T00:21:57.982 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Sep 17 00:21:57 smithi002 systemd[1]: Starting Ceph node-exporter.a for 3aad27a2-7489-11ef-bceb-c7b262605968... 2024-09-17T00:21:58.415 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Sep 17 00:21:58 smithi002 bash[57258]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-09-17T00:21:58.729 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Sep 17 00:21:58 smithi002 bash[57258]: Getting image source signatures 2024-09-17T00:21:58.729 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Sep 17 00:21:58 smithi002 bash[57258]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-09-17T00:21:58.729 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Sep 17 00:21:58 smithi002 bash[57258]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-09-17T00:21:58.729 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Sep 17 00:21:58 smithi002 bash[57258]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-09-17T00:21:59.479 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:21:59 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: ::ffff:172.21.15.99 - - [17/Sep/2024:00:21:59] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-09-17T00:21:59.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:21:59 smithi099 ceph-mon[24594]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-17T00:21:59.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:21:59 smithi002 ceph-mon[23737]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-17T00:21:59.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:21:59 smithi002 ceph-mon[28203]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-17T00:22:00.479 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Sep 17 00:22:00 smithi002 bash[57258]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-09-17T00:22:00.479 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Sep 17 00:22:00 smithi002 bash[57258]: Writing manifest to image destination 2024-09-17T00:22:00.855 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:00 smithi002 ceph-mon[23737]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T00:22:00.855 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:00 smithi002 ceph-mon[28203]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T00:22:00.856 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Sep 17 00:22:00 smithi002 podman[57258]: 2024-09-17 00:22:00.541414386 +0000 UTC m=+2.594278098 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-09-17T00:22:00.856 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Sep 17 00:22:00 smithi002 podman[57258]: 2024-09-17 00:22:00.631272461 +0000 UTC m=+2.684136171 container create 1030ae01bb32fb47fb23cd62a696b97e2d3be9b915056ff66a6f4a9ab8e90727 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-17T00:22:00.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:00 smithi099 ceph-mon[24594]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T00:22:01.118 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Sep 17 00:22:00 smithi002 podman[57258]: 2024-09-17 00:22:00.859011151 +0000 UTC m=+2.911874866 container init 1030ae01bb32fb47fb23cd62a696b97e2d3be9b915056ff66a6f4a9ab8e90727 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-17T00:22:01.118 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Sep 17 00:22:00 smithi002 podman[57258]: 2024-09-17 00:22:00.862458231 +0000 UTC m=+2.915321941 container start 1030ae01bb32fb47fb23cd62a696b97e2d3be9b915056ff66a6f4a9ab8e90727 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-17T00:22:01.118 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Sep 17 00:22:00 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-a[57334]: ts=2024-09-17T00:22:00.866Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-09-17T00:22:01.118 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Sep 17 00:22:00 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-a[57334]: ts=2024-09-17T00:22:00.866Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-09-17T00:22:01.118 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Sep 17 00:22:00 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-a[57334]: ts=2024-09-17T00:22:00.866Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-09-17T00:22:01.118 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Sep 17 00:22:00 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-a[57334]: ts=2024-09-17T00:22:00.866Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-09-17T00:22:01.119 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Sep 17 00:22:00 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-a[57334]: ts=2024-09-17T00:22:00.866Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-09-17T00:22:01.119 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Sep 17 00:22:00 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-a[57334]: ts=2024-09-17T00:22:00.866Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-09-17T00:22:01.119 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Sep 17 00:22:00 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-a[57334]: ts=2024-09-17T00:22:00.867Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-09-17T00:22:01.119 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Sep 17 00:22:00 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-a[57334]: ts=2024-09-17T00:22:00.867Z caller=node_exporter.go:117 level=info collector=arp 2024-09-17T00:22:01.119 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Sep 17 00:22:00 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-a[57334]: ts=2024-09-17T00:22:00.867Z caller=node_exporter.go:117 level=info collector=bcache 2024-09-17T00:22:01.119 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Sep 17 00:22:00 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-a[57334]: ts=2024-09-17T00:22:00.867Z caller=node_exporter.go:117 level=info collector=bonding 2024-09-17T00:22:01.119 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Sep 17 00:22:00 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-a[57334]: ts=2024-09-17T00:22:00.867Z caller=node_exporter.go:117 level=info collector=btrfs 2024-09-17T00:22:01.119 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Sep 17 00:22:00 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-a[57334]: ts=2024-09-17T00:22:00.867Z caller=node_exporter.go:117 level=info collector=conntrack 2024-09-17T00:22:01.119 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Sep 17 00:22:00 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-a[57334]: ts=2024-09-17T00:22:00.867Z caller=node_exporter.go:117 level=info collector=cpu 2024-09-17T00:22:01.119 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Sep 17 00:22:00 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-a[57334]: ts=2024-09-17T00:22:00.867Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-09-17T00:22:01.120 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Sep 17 00:22:00 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-a[57334]: ts=2024-09-17T00:22:00.867Z caller=node_exporter.go:117 level=info collector=diskstats 2024-09-17T00:22:01.120 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Sep 17 00:22:00 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-a[57334]: ts=2024-09-17T00:22:00.867Z caller=node_exporter.go:117 level=info collector=dmi 2024-09-17T00:22:01.120 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Sep 17 00:22:00 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-a[57334]: ts=2024-09-17T00:22:00.867Z caller=node_exporter.go:117 level=info collector=edac 2024-09-17T00:22:01.120 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Sep 17 00:22:00 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-a[57334]: ts=2024-09-17T00:22:00.867Z caller=node_exporter.go:117 level=info collector=entropy 2024-09-17T00:22:01.120 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Sep 17 00:22:00 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-a[57334]: ts=2024-09-17T00:22:00.867Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-09-17T00:22:01.120 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Sep 17 00:22:00 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-a[57334]: ts=2024-09-17T00:22:00.867Z caller=node_exporter.go:117 level=info collector=filefd 2024-09-17T00:22:01.120 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Sep 17 00:22:00 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-a[57334]: ts=2024-09-17T00:22:00.867Z caller=node_exporter.go:117 level=info collector=filesystem 2024-09-17T00:22:01.120 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Sep 17 00:22:00 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-a[57334]: ts=2024-09-17T00:22:00.867Z caller=node_exporter.go:117 level=info collector=hwmon 2024-09-17T00:22:01.120 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Sep 17 00:22:00 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-a[57334]: ts=2024-09-17T00:22:00.867Z caller=node_exporter.go:117 level=info collector=infiniband 2024-09-17T00:22:01.120 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Sep 17 00:22:00 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-a[57334]: ts=2024-09-17T00:22:00.867Z caller=node_exporter.go:117 level=info collector=ipvs 2024-09-17T00:22:01.120 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Sep 17 00:22:00 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-a[57334]: ts=2024-09-17T00:22:00.867Z caller=node_exporter.go:117 level=info collector=loadavg 2024-09-17T00:22:01.120 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Sep 17 00:22:00 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-a[57334]: ts=2024-09-17T00:22:00.867Z caller=node_exporter.go:117 level=info collector=mdadm 2024-09-17T00:22:01.121 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Sep 17 00:22:00 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-a[57334]: ts=2024-09-17T00:22:00.867Z caller=node_exporter.go:117 level=info collector=meminfo 2024-09-17T00:22:01.121 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Sep 17 00:22:00 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-a[57334]: ts=2024-09-17T00:22:00.867Z caller=node_exporter.go:117 level=info collector=netclass 2024-09-17T00:22:01.121 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Sep 17 00:22:00 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-a[57334]: ts=2024-09-17T00:22:00.867Z caller=node_exporter.go:117 level=info collector=netdev 2024-09-17T00:22:01.121 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Sep 17 00:22:00 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-a[57334]: ts=2024-09-17T00:22:00.867Z caller=node_exporter.go:117 level=info collector=netstat 2024-09-17T00:22:01.121 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Sep 17 00:22:00 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-a[57334]: ts=2024-09-17T00:22:00.867Z caller=node_exporter.go:117 level=info collector=nfs 2024-09-17T00:22:01.121 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Sep 17 00:22:00 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-a[57334]: ts=2024-09-17T00:22:00.867Z caller=node_exporter.go:117 level=info collector=nfsd 2024-09-17T00:22:01.121 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Sep 17 00:22:00 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-a[57334]: ts=2024-09-17T00:22:00.867Z caller=node_exporter.go:117 level=info collector=nvme 2024-09-17T00:22:01.121 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Sep 17 00:22:00 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-a[57334]: ts=2024-09-17T00:22:00.867Z caller=node_exporter.go:117 level=info collector=os 2024-09-17T00:22:01.121 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Sep 17 00:22:00 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-a[57334]: ts=2024-09-17T00:22:00.867Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-09-17T00:22:01.121 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Sep 17 00:22:00 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-a[57334]: ts=2024-09-17T00:22:00.867Z caller=node_exporter.go:117 level=info collector=pressure 2024-09-17T00:22:01.121 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Sep 17 00:22:00 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-a[57334]: ts=2024-09-17T00:22:00.867Z caller=node_exporter.go:117 level=info collector=rapl 2024-09-17T00:22:01.121 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Sep 17 00:22:00 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-a[57334]: ts=2024-09-17T00:22:00.867Z caller=node_exporter.go:117 level=info collector=schedstat 2024-09-17T00:22:01.121 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Sep 17 00:22:00 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-a[57334]: ts=2024-09-17T00:22:00.867Z caller=node_exporter.go:117 level=info collector=selinux 2024-09-17T00:22:01.121 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Sep 17 00:22:00 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-a[57334]: ts=2024-09-17T00:22:00.867Z caller=node_exporter.go:117 level=info collector=sockstat 2024-09-17T00:22:01.122 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Sep 17 00:22:00 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-a[57334]: ts=2024-09-17T00:22:00.867Z caller=node_exporter.go:117 level=info collector=softnet 2024-09-17T00:22:01.122 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Sep 17 00:22:00 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-a[57334]: ts=2024-09-17T00:22:00.867Z caller=node_exporter.go:117 level=info collector=stat 2024-09-17T00:22:01.122 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Sep 17 00:22:00 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-a[57334]: ts=2024-09-17T00:22:00.867Z caller=node_exporter.go:117 level=info collector=tapestats 2024-09-17T00:22:01.122 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Sep 17 00:22:00 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-a[57334]: ts=2024-09-17T00:22:00.867Z caller=node_exporter.go:117 level=info collector=textfile 2024-09-17T00:22:01.122 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Sep 17 00:22:00 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-a[57334]: ts=2024-09-17T00:22:00.867Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-09-17T00:22:01.122 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Sep 17 00:22:00 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-a[57334]: ts=2024-09-17T00:22:00.867Z caller=node_exporter.go:117 level=info collector=time 2024-09-17T00:22:01.122 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Sep 17 00:22:00 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-a[57334]: ts=2024-09-17T00:22:00.867Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-09-17T00:22:01.122 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Sep 17 00:22:00 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-a[57334]: ts=2024-09-17T00:22:00.867Z caller=node_exporter.go:117 level=info collector=uname 2024-09-17T00:22:01.122 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Sep 17 00:22:00 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-a[57334]: ts=2024-09-17T00:22:00.867Z caller=node_exporter.go:117 level=info collector=vmstat 2024-09-17T00:22:01.122 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Sep 17 00:22:00 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-a[57334]: ts=2024-09-17T00:22:00.867Z caller=node_exporter.go:117 level=info collector=xfs 2024-09-17T00:22:01.122 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Sep 17 00:22:00 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-a[57334]: ts=2024-09-17T00:22:00.867Z caller=node_exporter.go:117 level=info collector=zfs 2024-09-17T00:22:01.122 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Sep 17 00:22:00 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-a[57334]: ts=2024-09-17T00:22:00.868Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-09-17T00:22:01.122 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Sep 17 00:22:00 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-a[57334]: ts=2024-09-17T00:22:00.868Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-09-17T00:22:01.122 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Sep 17 00:22:00 smithi002 bash[57258]: 1030ae01bb32fb47fb23cd62a696b97e2d3be9b915056ff66a6f4a9ab8e90727 2024-09-17T00:22:01.123 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Sep 17 00:22:00 smithi002 systemd[1]: Started Ceph node-exporter.a for 3aad27a2-7489-11ef-bceb-c7b262605968. 2024-09-17T00:22:01.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:01 smithi099 ceph-mon[24594]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-17T00:22:01.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:01 smithi099 ceph-mon[24594]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:22:01.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:01 smithi099 ceph-mon[24594]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:22:01.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:01 smithi099 ceph-mon[24594]: Reconfiguring alertmanager.a (dependencies changed)... 2024-09-17T00:22:01.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:01 smithi099 ceph-mon[24594]: Deploying daemon alertmanager.a on smithi002 2024-09-17T00:22:01.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:01 smithi002 ceph-mon[28203]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-17T00:22:01.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:01 smithi002 ceph-mon[28203]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:22:01.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:01 smithi002 ceph-mon[28203]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:22:01.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:01 smithi002 ceph-mon[28203]: Reconfiguring alertmanager.a (dependencies changed)... 2024-09-17T00:22:01.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:01 smithi002 ceph-mon[28203]: Deploying daemon alertmanager.a on smithi002 2024-09-17T00:22:01.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:01 smithi002 ceph-mon[23737]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-17T00:22:01.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:01 smithi002 ceph-mon[23737]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:22:01.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:01 smithi002 ceph-mon[23737]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:22:01.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:01 smithi002 ceph-mon[23737]: Reconfiguring alertmanager.a (dependencies changed)... 2024-09-17T00:22:01.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:01 smithi002 ceph-mon[23737]: Deploying daemon alertmanager.a on smithi002 2024-09-17T00:22:03.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:03 smithi099 ceph-mon[24594]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-17T00:22:03.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:03 smithi002 ceph-mon[23737]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-17T00:22:03.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:03 smithi002 ceph-mon[28203]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-17T00:22:03.979 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Sep 17 00:22:03 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-alertmanager-a[48044]: level=error ts=2024-09-17T00:22:03.509Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.99:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.99 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.2:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.2 because it doesn't contain any IP SANs" 2024-09-17T00:22:03.980 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Sep 17 00:22:03 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-alertmanager-a[48044]: level=warn ts=2024-09-17T00:22:03.511Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.2:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.2 because it doesn't contain any IP SANs" 2024-09-17T00:22:03.980 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Sep 17 00:22:03 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-alertmanager-a[48044]: level=warn ts=2024-09-17T00:22:03.511Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.99:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.99 because it doesn't contain any IP SANs" 2024-09-17T00:22:04.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:04 smithi099 ceph-mon[24594]: from='client.14889 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:22:04.967 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:22:04 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: ::ffff:172.21.15.99 - - [17/Sep/2024:00:22:04] "GET /metrics HTTP/1.1" 200 38135 "" "Prometheus/2.33.4" 2024-09-17T00:22:04.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:04 smithi002 ceph-mon[28203]: from='client.14889 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:22:04.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:04 smithi002 ceph-mon[23737]: from='client.14889 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:22:05.537 INFO:teuthology.orchestra.run.smithi002.stdout:true 2024-09-17T00:22:05.966 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:05 smithi099 ceph-mon[24594]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:22:05.981 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:05 smithi002 ceph-mon[28203]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:22:05.981 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:05 smithi002 ceph-mon[23737]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:22:06.237 INFO:teuthology.orchestra.run.smithi002.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-17T00:22:06.237 INFO:teuthology.orchestra.run.smithi002.stdout:alertmanager.a smithi002 *:9093,9094 running (4m) 15s ago 4m 26.9M - 0.23.0 ba2b418f427c f8bd5774d324 2024-09-17T00:22:06.237 INFO:teuthology.orchestra.run.smithi002.stdout:grafana.a smithi099 *:3000 running (4m) 15s ago 4m 55.5M - 8.3.5 dad864ee21e9 1547cd171832 2024-09-17T00:22:06.237 INFO:teuthology.orchestra.run.smithi002.stdout:iscsi.foo.smithi002.vmouem smithi002 running (3m) 15s ago 3m 52.1M - 3.5 e1d6a67b021e 6b47041ec00f 2024-09-17T00:22:06.237 INFO:teuthology.orchestra.run.smithi002.stdout:mgr.x smithi099 *:8443,9283 running (30s) 15s ago 9m 481M - 19.1.1-330-gc2ddf972 ebb0615149de a8e1e8964c84 2024-09-17T00:22:06.237 INFO:teuthology.orchestra.run.smithi002.stdout:mgr.y smithi002 *:9283 running (11m) 15s ago 11m 173M - 17.2.0 e1d6a67b021e d32b04d294d3 2024-09-17T00:22:06.237 INFO:teuthology.orchestra.run.smithi002.stdout:mon.a smithi002 running (11m) 15s ago 11m 54.2M 2048M 17.2.0 e1d6a67b021e a68823df625f 2024-09-17T00:22:06.237 INFO:teuthology.orchestra.run.smithi002.stdout:mon.b smithi099 running (9m) 15s ago 9m 44.2M 2048M 17.2.0 e1d6a67b021e c6d2af715187 2024-09-17T00:22:06.237 INFO:teuthology.orchestra.run.smithi002.stdout:mon.c smithi002 running (9m) 15s ago 9m 41.1M 2048M 17.2.0 e1d6a67b021e 77b3d2e14899 2024-09-17T00:22:06.237 INFO:teuthology.orchestra.run.smithi002.stdout:node-exporter.a smithi002 *:9100 starting - - - - 2024-09-17T00:22:06.238 INFO:teuthology.orchestra.run.smithi002.stdout:node-exporter.b smithi099 *:9100 running (5m) 15s ago 5m 17.2M - 1.3.1 1dbe0e931976 352ae73fe2a7 2024-09-17T00:22:06.238 INFO:teuthology.orchestra.run.smithi002.stdout:osd.0 smithi002 running (9m) 15s ago 9m 52.4M 2737M 17.2.0 e1d6a67b021e a0ad0fb7aa34 2024-09-17T00:22:06.238 INFO:teuthology.orchestra.run.smithi002.stdout:osd.1 smithi002 running (8m) 15s ago 8m 54.5M 2737M 17.2.0 e1d6a67b021e 7cb62c311c7d 2024-09-17T00:22:06.238 INFO:teuthology.orchestra.run.smithi002.stdout:osd.2 smithi002 running (8m) 15s ago 8m 50.7M 2737M 17.2.0 e1d6a67b021e eafddacb955e 2024-09-17T00:22:06.238 INFO:teuthology.orchestra.run.smithi002.stdout:osd.3 smithi002 running (7m) 15s ago 7m 54.8M 2737M 17.2.0 e1d6a67b021e fd99d3764412 2024-09-17T00:22:06.238 INFO:teuthology.orchestra.run.smithi002.stdout:osd.4 smithi099 running (7m) 15s ago 7m 54.5M 2992M 17.2.0 e1d6a67b021e 0af1827b658d 2024-09-17T00:22:06.238 INFO:teuthology.orchestra.run.smithi002.stdout:osd.5 smithi099 running (7m) 15s ago 7m 53.2M 2992M 17.2.0 e1d6a67b021e 4f7ef3aefc58 2024-09-17T00:22:06.238 INFO:teuthology.orchestra.run.smithi002.stdout:osd.6 smithi099 running (6m) 15s ago 6m 49.4M 2992M 17.2.0 e1d6a67b021e 3213f5e6952a 2024-09-17T00:22:06.238 INFO:teuthology.orchestra.run.smithi002.stdout:osd.7 smithi099 running (6m) 15s ago 6m 55.5M 2992M 17.2.0 e1d6a67b021e 63be3a1ed0a1 2024-09-17T00:22:06.238 INFO:teuthology.orchestra.run.smithi002.stdout:prometheus.a smithi099 *:9095 running (4m) 15s ago 5m 67.0M - 2.33.4 514e6a882f6e a1529c2c7a1c 2024-09-17T00:22:06.238 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.foo.smithi002.uwdhcs smithi002 *:8000 running (4m) 15s ago 4m 73.3M - 17.2.0 e1d6a67b021e 9c35813f459a 2024-09-17T00:22:06.238 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.foo.smithi099.utzfap smithi099 *:8000 running (4m) 15s ago 4m 73.6M - 17.2.0 e1d6a67b021e 34ec989160a6 2024-09-17T00:22:06.238 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.smpl.smithi002.rqnxii smithi002 *:80 running (3m) 15s ago 3m 71.6M - 17.2.0 e1d6a67b021e 0e180d0afb99 2024-09-17T00:22:06.238 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.smpl.smithi099.nzvkpm smithi099 *:80 running (3m) 15s ago 3m 74.4M - 17.2.0 e1d6a67b021e 6a068e82ac72 2024-09-17T00:22:06.651 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-09-17T00:22:06.652 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": { 2024-09-17T00:22:06.652 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-09-17T00:22:06.652 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-09-17T00:22:06.652 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": { 2024-09-17T00:22:06.652 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-09-17T00:22:06.652 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 19.1.1-330-gc2ddf972 (c2ddf9721ef254645201cdbabd9c8f49c30a586f) squid (stable)": 1 2024-09-17T00:22:06.652 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-09-17T00:22:06.652 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": { 2024-09-17T00:22:06.652 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-09-17T00:22:06.652 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-09-17T00:22:06.652 INFO:teuthology.orchestra.run.smithi002.stdout: "mds": {}, 2024-09-17T00:22:06.652 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw": { 2024-09-17T00:22:06.652 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-09-17T00:22:06.653 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-09-17T00:22:06.653 INFO:teuthology.orchestra.run.smithi002.stdout: "overall": { 2024-09-17T00:22:06.653 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 16, 2024-09-17T00:22:06.653 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 19.1.1-330-gc2ddf972 (c2ddf9721ef254645201cdbabd9c8f49c30a586f) squid (stable)": 1 2024-09-17T00:22:06.653 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-09-17T00:22:06.653 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-09-17T00:22:06.838 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:06 smithi002 ceph-mon[28203]: from='client.15195 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:22:06.838 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:06 smithi002 ceph-mon[28203]: from='client.? 172.21.15.2:0/312306252' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:22:06.838 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:06 smithi002 ceph-mon[23737]: from='client.15195 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:22:06.838 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:06 smithi002 ceph-mon[23737]: from='client.? 172.21.15.2:0/312306252' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:22:06.966 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:06 smithi099 ceph-mon[24594]: from='client.15195 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:22:06.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:06 smithi099 ceph-mon[24594]: from='client.? 172.21.15.2:0/312306252' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:22:07.007 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-09-17T00:22:07.008 INFO:teuthology.orchestra.run.smithi002.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f", 2024-09-17T00:22:07.008 INFO:teuthology.orchestra.run.smithi002.stdout: "in_progress": true, 2024-09-17T00:22:07.008 INFO:teuthology.orchestra.run.smithi002.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-17T00:22:07.008 INFO:teuthology.orchestra.run.smithi002.stdout: "services_complete": [], 2024-09-17T00:22:07.008 INFO:teuthology.orchestra.run.smithi002.stdout: "progress": "1/23 daemons upgraded", 2024-09-17T00:22:07.008 INFO:teuthology.orchestra.run.smithi002.stdout: "message": "", 2024-09-17T00:22:07.008 INFO:teuthology.orchestra.run.smithi002.stdout: "is_paused": false 2024-09-17T00:22:07.008 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-09-17T00:22:07.435 INFO:teuthology.orchestra.run.smithi002.stdout:HEALTH_OK 2024-09-17T00:22:07.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:07 smithi099 ceph-mon[24594]: from='client.15201 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:22:07.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:07 smithi099 ceph-mon[24594]: from='client.25093 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:22:07.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:07 smithi099 ceph-mon[24594]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:22:07.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:07 smithi099 ceph-mon[24594]: from='client.? 172.21.15.2:0/4097312620' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T00:22:07.977 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:07 smithi002 ceph-mon[23737]: from='client.15201 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:22:07.977 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:07 smithi002 ceph-mon[23737]: from='client.25093 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:22:07.977 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:07 smithi002 ceph-mon[23737]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:22:07.977 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:07 smithi002 ceph-mon[23737]: from='client.? 172.21.15.2:0/4097312620' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T00:22:07.978 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:07 smithi002 ceph-mon[28203]: from='client.15201 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:22:07.978 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:07 smithi002 ceph-mon[28203]: from='client.25093 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:22:07.978 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:07 smithi002 ceph-mon[28203]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:22:07.978 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:07 smithi002 ceph-mon[28203]: from='client.? 172.21.15.2:0/4097312620' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T00:22:08.625 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Sep 17 00:22:08 smithi002 systemd[1]: Stopping Ceph alertmanager.a for 3aad27a2-7489-11ef-bceb-c7b262605968... 2024-09-17T00:22:08.942 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:08 smithi002 ceph-mon[23737]: from='client.25105 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:22:08.943 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:08 smithi002 ceph-mon[28203]: from='client.25105 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:22:08.943 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Sep 17 00:22:08 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-alertmanager-a[48044]: level=info ts=2024-09-17T00:22:08.628Z caller=main.go:557 msg="Received SIGTERM, exiting gracefully..." 2024-09-17T00:22:08.943 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Sep 17 00:22:08 smithi002 podman[57839]: 2024-09-17 00:22:08.6492378 +0000 UTC m=+0.126402593 container died f8bd5774d3248d7796a45c11b454ab89f567b72b0bc0b7ed04f7eeb2e24f06ed (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-17T00:22:08.966 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:08 smithi099 ceph-mon[24594]: from='client.25105 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:22:09.229 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:22:09 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: ::ffff:172.21.15.99 - - [17/Sep/2024:00:22:09] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-09-17T00:22:09.229 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Sep 17 00:22:09 smithi002 podman[57839]: 2024-09-17 00:22:09.15366596 +0000 UTC m=+0.630830786 container cleanup f8bd5774d3248d7796a45c11b454ab89f567b72b0bc0b7ed04f7eeb2e24f06ed (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-17T00:22:09.229 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Sep 17 00:22:09 smithi002 bash[57839]: ceph-3aad27a2-7489-11ef-bceb-c7b262605968-alertmanager-a 2024-09-17T00:22:09.636 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Sep 17 00:22:09 smithi002 podman[57850]: 2024-09-17 00:22:09.380228194 +0000 UTC m=+0.725633653 container remove f8bd5774d3248d7796a45c11b454ab89f567b72b0bc0b7ed04f7eeb2e24f06ed (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-17T00:22:09.636 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Sep 17 00:22:09 smithi002 podman[57850]: 2024-09-17 00:22:09.464483893 +0000 UTC m=+0.809889308 volume remove e47b4d4b018bf8c90622b1043393783f76c44a782c3d54c4f43f9f264f2172cf 2024-09-17T00:22:09.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:09 smithi099 ceph-mon[24594]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:22:09.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:09 smithi002 ceph-mon[23737]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:22:09.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:09 smithi002 ceph-mon[28203]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:22:09.979 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Sep 17 00:22:09 smithi002 systemd[1]: ceph-3aad27a2-7489-11ef-bceb-c7b262605968@alertmanager.a.service: Deactivated successfully. 2024-09-17T00:22:09.979 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Sep 17 00:22:09 smithi002 systemd[1]: Stopped Ceph alertmanager.a for 3aad27a2-7489-11ef-bceb-c7b262605968. 2024-09-17T00:22:09.979 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Sep 17 00:22:09 smithi002 systemd[1]: ceph-3aad27a2-7489-11ef-bceb-c7b262605968@alertmanager.a.service: Consumed 1.260s CPU time. 2024-09-17T00:22:09.979 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Sep 17 00:22:09 smithi002 systemd[1]: Starting Ceph alertmanager.a for 3aad27a2-7489-11ef-bceb-c7b262605968... 2024-09-17T00:22:10.362 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Sep 17 00:22:10 smithi002 podman[57985]: 2024-09-17 00:22:09.974440292 +0000 UTC m=+0.016795052 image pull c8568f914cd25b2062c44e9f79f9c18da6e3b85fe0c47a12a2191c61426c2b19 quay.io/prometheus/alertmanager:v0.25.0 2024-09-17T00:22:10.362 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Sep 17 00:22:10 smithi002 podman[57985]: 2024-09-17 00:22:10.090335957 +0000 UTC m=+0.132690715 volume create 3a9c87d8fad1e17a1537f1b96110a81e6e678e5783bd08b2b078ebed1ef087cd 2024-09-17T00:22:10.362 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Sep 17 00:22:10 smithi002 podman[57985]: 2024-09-17 00:22:10.182224134 +0000 UTC m=+0.224578892 container create a21e13b3d67719c3fd78a1a24cfb5cd52ae1bbafd1a6624342f9d7028331d94c (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-17T00:22:10.729 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Sep 17 00:22:10 smithi002 podman[57985]: 2024-09-17 00:22:10.468520538 +0000 UTC m=+0.510875312 container init a21e13b3d67719c3fd78a1a24cfb5cd52ae1bbafd1a6624342f9d7028331d94c (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-17T00:22:10.729 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Sep 17 00:22:10 smithi002 podman[57985]: 2024-09-17 00:22:10.471697327 +0000 UTC m=+0.514052092 container start a21e13b3d67719c3fd78a1a24cfb5cd52ae1bbafd1a6624342f9d7028331d94c (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-17T00:22:10.729 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Sep 17 00:22:10 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-alertmanager-a[58000]: ts=2024-09-17T00:22:10.486Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-09-17T00:22:10.729 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Sep 17 00:22:10 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-alertmanager-a[58000]: ts=2024-09-17T00:22:10.486Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-09-17T00:22:10.730 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Sep 17 00:22:10 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-alertmanager-a[58000]: ts=2024-09-17T00:22:10.487Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.2 port=9094 2024-09-17T00:22:10.730 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Sep 17 00:22:10 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-alertmanager-a[58000]: ts=2024-09-17T00:22:10.488Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-09-17T00:22:10.730 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Sep 17 00:22:10 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-alertmanager-a[58000]: ts=2024-09-17T00:22:10.513Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-17T00:22:10.730 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Sep 17 00:22:10 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-alertmanager-a[58000]: ts=2024-09-17T00:22:10.513Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-17T00:22:10.730 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Sep 17 00:22:10 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-alertmanager-a[58000]: ts=2024-09-17T00:22:10.516Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-09-17T00:22:10.730 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Sep 17 00:22:10 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-alertmanager-a[58000]: ts=2024-09-17T00:22:10.516Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-09-17T00:22:10.730 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Sep 17 00:22:10 smithi002 bash[57985]: a21e13b3d67719c3fd78a1a24cfb5cd52ae1bbafd1a6624342f9d7028331d94c 2024-09-17T00:22:10.730 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Sep 17 00:22:10 smithi002 systemd[1]: Started Ceph alertmanager.a for 3aad27a2-7489-11ef-bceb-c7b262605968. 2024-09-17T00:22:11.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:11 smithi099 ceph-mon[24594]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:22:11.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:11 smithi099 ceph-mon[24594]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:22:11.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:11 smithi099 ceph-mon[24594]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:22:11.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:11 smithi099 ceph-mon[24594]: Reconfiguring iscsi.foo.smithi002.vmouem (dependencies changed)... 2024-09-17T00:22:11.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:11 smithi099 ceph-mon[24594]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi002.vmouem", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-17T00:22:11.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:11 smithi099 ceph-mon[24594]: from='mgr.24986 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi002.vmouem", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-17T00:22:11.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:11 smithi099 ceph-mon[24594]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:22:11.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:11 smithi099 ceph-mon[24594]: Reconfiguring daemon iscsi.foo.smithi002.vmouem on smithi002 2024-09-17T00:22:11.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:11 smithi002 ceph-mon[28203]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:22:11.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:11 smithi002 ceph-mon[28203]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:22:11.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:11 smithi002 ceph-mon[28203]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:22:11.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:11 smithi002 ceph-mon[28203]: Reconfiguring iscsi.foo.smithi002.vmouem (dependencies changed)... 2024-09-17T00:22:11.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:11 smithi002 ceph-mon[28203]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi002.vmouem", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-17T00:22:11.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:11 smithi002 ceph-mon[28203]: from='mgr.24986 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi002.vmouem", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-17T00:22:11.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:11 smithi002 ceph-mon[28203]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:22:11.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:11 smithi002 ceph-mon[28203]: Reconfiguring daemon iscsi.foo.smithi002.vmouem on smithi002 2024-09-17T00:22:11.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:11 smithi002 ceph-mon[23737]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:22:11.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:11 smithi002 ceph-mon[23737]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:22:11.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:11 smithi002 ceph-mon[23737]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:22:11.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:11 smithi002 ceph-mon[23737]: Reconfiguring iscsi.foo.smithi002.vmouem (dependencies changed)... 2024-09-17T00:22:11.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:11 smithi002 ceph-mon[23737]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi002.vmouem", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-17T00:22:11.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:11 smithi002 ceph-mon[23737]: from='mgr.24986 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi002.vmouem", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-17T00:22:11.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:11 smithi002 ceph-mon[23737]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:22:11.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:11 smithi002 ceph-mon[23737]: Reconfiguring daemon iscsi.foo.smithi002.vmouem on smithi002 2024-09-17T00:22:12.717 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:22:12 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[41464]: ts=2024-09-17T00:22:12.314Z caller=manager.go:609 level=warn component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.99:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.2:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-17T00:22:12.871 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Sep 17 00:22:12 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-alertmanager-a[58000]: ts=2024-09-17T00:22:12.489Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000551876s 2024-09-17T00:22:13.968 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:13 smithi099 ceph-mon[24594]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:22:13.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:13 smithi002 ceph-mon[23737]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:22:13.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:13 smithi002 ceph-mon[28203]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:22:14.467 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:22:13 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: 2024-09-17T00:22:13.972+0000 7f0db517b640 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi002.vmouem ... 2024-09-17T00:22:14.467 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:22:13 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: Creating ceph-iscsi config... 2024-09-17T00:22:14.468 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:22:13 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/iscsi-gateway.cfg 2024-09-17T00:22:14.468 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:22:13 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/tcmu-runner-entrypoint.sh 2024-09-17T00:22:14.468 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:22:13 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: Non-zero exit code 1 from systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem 2024-09-17T00:22:14.468 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:22:13 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: systemctl: stderr Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:22:14.468 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:22:13 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: systemctl: stderr See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:22:14.468 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:22:13 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: Traceback (most recent call last): 2024-09-17T00:22:14.468 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:22:13 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-17T00:22:14.468 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:22:13 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: return _run_code(code, main_globals, None, 2024-09-17T00:22:14.468 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:22:13 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-17T00:22:14.468 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:22:13 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: exec(code, run_globals) 2024-09-17T00:22:14.468 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:22:13 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5579, in 2024-09-17T00:22:14.468 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:22:13 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5567, in main 2024-09-17T00:22:14.469 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:22:13 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3049, in command_deploy_from 2024-09-17T00:22:14.469 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:22:13 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3084, in _common_deploy 2024-09-17T00:22:14.469 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:22:13 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3104, in _deploy_daemon_container 2024-09-17T00:22:14.469 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:22:13 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 1124, in deploy_daemon 2024-09-17T00:22:14.469 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:22:13 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: File "/tmp/tmpnroy8d0w.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-17T00:22:14.469 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:22:13 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: RuntimeError: Failed command: systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem: Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:22:14.469 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:22:13 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:22:14.469 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:22:13 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: Traceback (most recent call last): 2024-09-17T00:22:14.470 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:22:13 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1134, in _check_daemons 2024-09-17T00:22:14.470 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:22:13 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: self.mgr._daemon_action(daemon_spec, action=action) 2024-09-17T00:22:14.470 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:22:13 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2473, in _daemon_action 2024-09-17T00:22:14.470 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:22:13 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: return self.wait_async( 2024-09-17T00:22:14.470 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:22:13 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: File "/usr/share/ceph/mgr/cephadm/module.py", line 796, in wait_async 2024-09-17T00:22:14.470 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:22:13 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: return self.event_loop.get_result(coro, timeout) 2024-09-17T00:22:14.470 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:22:13 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-09-17T00:22:14.470 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:22:13 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: return future.result(timeout) 2024-09-17T00:22:14.470 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:22:13 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: File "/lib64/python3.9/concurrent/futures/_base.py", line 446, in result 2024-09-17T00:22:14.470 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:22:13 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: return self.__get_result() 2024-09-17T00:22:14.470 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:22:13 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: File "/lib64/python3.9/concurrent/futures/_base.py", line 391, in __get_result 2024-09-17T00:22:14.470 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:22:13 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: raise self._exception 2024-09-17T00:22:14.471 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:22:13 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1370, in _create_daemon 2024-09-17T00:22:14.471 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:22:13 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: out, err, code = await self._run_cephadm( 2024-09-17T00:22:14.471 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:22:13 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1713, in _run_cephadm 2024-09-17T00:22:14.471 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:22:13 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: raise OrchestratorError( 2024-09-17T00:22:14.471 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:22:13 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi002.vmouem ... 2024-09-17T00:22:14.471 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:22:13 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: Creating ceph-iscsi config... 2024-09-17T00:22:14.471 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:22:13 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/iscsi-gateway.cfg 2024-09-17T00:22:14.471 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:22:13 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/tcmu-runner-entrypoint.sh 2024-09-17T00:22:14.471 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:22:13 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: Non-zero exit code 1 from systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem 2024-09-17T00:22:14.471 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:22:13 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: systemctl: stderr Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:22:14.471 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:22:13 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: systemctl: stderr See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:22:14.471 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:22:13 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: Traceback (most recent call last): 2024-09-17T00:22:14.471 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:22:13 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-17T00:22:14.471 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:22:13 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: return _run_code(code, main_globals, None, 2024-09-17T00:22:14.472 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:22:13 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-17T00:22:14.472 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:22:13 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: exec(code, run_globals) 2024-09-17T00:22:14.472 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:22:13 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5579, in 2024-09-17T00:22:14.472 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:22:13 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5567, in main 2024-09-17T00:22:14.472 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:22:13 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3049, in command_deploy_from 2024-09-17T00:22:14.472 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:22:13 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3084, in _common_deploy 2024-09-17T00:22:14.472 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:22:13 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3104, in _deploy_daemon_container 2024-09-17T00:22:14.472 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:22:13 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 1124, in deploy_daemon 2024-09-17T00:22:14.472 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:22:13 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: File "/tmp/tmpnroy8d0w.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-17T00:22:14.472 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:22:13 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: RuntimeError: Failed command: systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem: Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:22:14.472 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:22:13 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:22:15.216 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:22:14 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: ::ffff:172.21.15.99 - - [17/Sep/2024:00:22:14] "GET /metrics HTTP/1.1" 200 38135 "" "Prometheus/2.33.4" 2024-09-17T00:22:15.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:15 smithi099 ceph-mon[24594]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi002.vmouem ... 2024-09-17T00:22:15.967 INFO:journalctl@ceph.mon.b.smithi099.stdout: Creating ceph-iscsi config... 2024-09-17T00:22:15.967 INFO:journalctl@ceph.mon.b.smithi099.stdout: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/iscsi-gateway.cfg 2024-09-17T00:22:15.967 INFO:journalctl@ceph.mon.b.smithi099.stdout: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/tcmu-runner-entrypoint.sh 2024-09-17T00:22:15.967 INFO:journalctl@ceph.mon.b.smithi099.stdout: Non-zero exit code 1 from systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem 2024-09-17T00:22:15.967 INFO:journalctl@ceph.mon.b.smithi099.stdout: systemctl: stderr Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:22:15.967 INFO:journalctl@ceph.mon.b.smithi099.stdout: systemctl: stderr See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:22:15.968 INFO:journalctl@ceph.mon.b.smithi099.stdout: Traceback (most recent call last): 2024-09-17T00:22:15.968 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-17T00:22:15.968 INFO:journalctl@ceph.mon.b.smithi099.stdout: return _run_code(code, main_globals, None, 2024-09-17T00:22:15.968 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-17T00:22:15.968 INFO:journalctl@ceph.mon.b.smithi099.stdout: exec(code, run_globals) 2024-09-17T00:22:15.968 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5579, in 2024-09-17T00:22:15.968 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5567, in main 2024-09-17T00:22:15.968 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3049, in command_deploy_from 2024-09-17T00:22:15.968 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3084, in _common_deploy 2024-09-17T00:22:15.968 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3104, in _deploy_daemon_container 2024-09-17T00:22:15.968 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 1124, in deploy_daemon 2024-09-17T00:22:15.968 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-17T00:22:15.968 INFO:journalctl@ceph.mon.b.smithi099.stdout: RuntimeError: Failed command: systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem: Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:22:15.968 INFO:journalctl@ceph.mon.b.smithi099.stdout: See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:22:15.969 INFO:journalctl@ceph.mon.b.smithi099.stdout: Traceback (most recent call last): 2024-09-17T00:22:15.969 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1134, in _check_daemons 2024-09-17T00:22:15.969 INFO:journalctl@ceph.mon.b.smithi099.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-09-17T00:22:15.969 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2473, in _daemon_action 2024-09-17T00:22:15.969 INFO:journalctl@ceph.mon.b.smithi099.stdout: return self.wait_async( 2024-09-17T00:22:15.969 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 796, in wait_async 2024-09-17T00:22:15.969 INFO:journalctl@ceph.mon.b.smithi099.stdout: return self.event_loop.get_result(coro, timeout) 2024-09-17T00:22:15.969 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-09-17T00:22:15.969 INFO:journalctl@ceph.mon.b.smithi099.stdout: return future.result(timeout) 2024-09-17T00:22:15.969 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/lib64/python3.9/concurrent/futures/_base.py", line 446, in result 2024-09-17T00:22:15.969 INFO:journalctl@ceph.mon.b.smithi099.stdout: return self.__get_result() 2024-09-17T00:22:15.969 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/lib64/python3.9/concurrent/futures/_base.py", line 391, in __get_result 2024-09-17T00:22:15.969 INFO:journalctl@ceph.mon.b.smithi099.stdout: raise self._exception 2024-09-17T00:22:15.969 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1370, in _create_daemon 2024-09-17T00:22:15.970 INFO:journalctl@ceph.mon.b.smithi099.stdout: out, err, code = await self._run_cephadm( 2024-09-17T00:22:15.970 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1713, in _run_cephadm 2024-09-17T00:22:15.970 INFO:journalctl@ceph.mon.b.smithi099.stdout: raise OrchestratorError( 2024-09-17T00:22:15.970 INFO:journalctl@ceph.mon.b.smithi099.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi002.vmouem ... 2024-09-17T00:22:15.970 INFO:journalctl@ceph.mon.b.smithi099.stdout: Creating ceph-iscsi config... 2024-09-17T00:22:15.970 INFO:journalctl@ceph.mon.b.smithi099.stdout: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/iscsi-gateway.cfg 2024-09-17T00:22:15.970 INFO:journalctl@ceph.mon.b.smithi099.stdout: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/tcmu-runner-entrypoint.sh 2024-09-17T00:22:15.970 INFO:journalctl@ceph.mon.b.smithi099.stdout: Non-zero exit code 1 from systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem 2024-09-17T00:22:15.970 INFO:journalctl@ceph.mon.b.smithi099.stdout: systemctl: stderr Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:22:15.970 INFO:journalctl@ceph.mon.b.smithi099.stdout: systemctl: stderr See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:22:15.970 INFO:journalctl@ceph.mon.b.smithi099.stdout: Traceback (most recent call last): 2024-09-17T00:22:15.970 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-17T00:22:15.970 INFO:journalctl@ceph.mon.b.smithi099.stdout: return _run_code(code, main_globals, None, 2024-09-17T00:22:15.970 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-17T00:22:15.971 INFO:journalctl@ceph.mon.b.smithi099.stdout: exec(code, run_globals) 2024-09-17T00:22:15.971 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5579, in 2024-09-17T00:22:15.971 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5567, in main 2024-09-17T00:22:15.971 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3049, in command_deploy_from 2024-09-17T00:22:15.971 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3084, in _common_deploy 2024-09-17T00:22:15.971 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3104, in _deploy_daemon_container 2024-09-17T00:22:15.971 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 1124, in deploy_daemon 2024-09-17T00:22:15.971 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-17T00:22:15.971 INFO:journalctl@ceph.mon.b.smithi099.stdout: RuntimeError: Failed command: systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem: Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:22:15.971 INFO:journalctl@ceph.mon.b.smithi099.stdout: See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:22:15.971 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:15 smithi099 ceph-mon[24594]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-17T00:22:15.971 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:15 smithi099 ceph-mon[24594]: Deploying daemon prometheus.a on smithi099 2024-09-17T00:22:15.971 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:15 smithi099 ceph-mon[24594]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:22:15.971 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:15 smithi099 ceph-mon[24594]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T00:22:15.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:15 smithi002 ceph-mon[23737]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi002.vmouem ... 2024-09-17T00:22:15.979 INFO:journalctl@ceph.mon.a.smithi002.stdout: Creating ceph-iscsi config... 2024-09-17T00:22:15.979 INFO:journalctl@ceph.mon.a.smithi002.stdout: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/iscsi-gateway.cfg 2024-09-17T00:22:15.979 INFO:journalctl@ceph.mon.a.smithi002.stdout: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/tcmu-runner-entrypoint.sh 2024-09-17T00:22:15.979 INFO:journalctl@ceph.mon.a.smithi002.stdout: Non-zero exit code 1 from systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem 2024-09-17T00:22:15.979 INFO:journalctl@ceph.mon.a.smithi002.stdout: systemctl: stderr Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:22:15.979 INFO:journalctl@ceph.mon.a.smithi002.stdout: systemctl: stderr See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:22:15.979 INFO:journalctl@ceph.mon.a.smithi002.stdout: Traceback (most recent call last): 2024-09-17T00:22:15.980 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-17T00:22:15.980 INFO:journalctl@ceph.mon.a.smithi002.stdout: return _run_code(code, main_globals, None, 2024-09-17T00:22:15.980 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-17T00:22:15.980 INFO:journalctl@ceph.mon.a.smithi002.stdout: exec(code, run_globals) 2024-09-17T00:22:15.980 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5579, in 2024-09-17T00:22:15.980 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5567, in main 2024-09-17T00:22:15.980 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3049, in command_deploy_from 2024-09-17T00:22:15.980 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3084, in _common_deploy 2024-09-17T00:22:15.980 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3104, in _deploy_daemon_container 2024-09-17T00:22:15.980 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 1124, in deploy_daemon 2024-09-17T00:22:15.980 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-17T00:22:15.980 INFO:journalctl@ceph.mon.a.smithi002.stdout: RuntimeError: Failed command: systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem: Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:22:15.980 INFO:journalctl@ceph.mon.a.smithi002.stdout: See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:22:15.980 INFO:journalctl@ceph.mon.a.smithi002.stdout: Traceback (most recent call last): 2024-09-17T00:22:15.981 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1134, in _check_daemons 2024-09-17T00:22:15.981 INFO:journalctl@ceph.mon.a.smithi002.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-09-17T00:22:15.981 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2473, in _daemon_action 2024-09-17T00:22:15.981 INFO:journalctl@ceph.mon.a.smithi002.stdout: return self.wait_async( 2024-09-17T00:22:15.981 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 796, in wait_async 2024-09-17T00:22:15.981 INFO:journalctl@ceph.mon.a.smithi002.stdout: return self.event_loop.get_result(coro, timeout) 2024-09-17T00:22:15.981 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-09-17T00:22:15.981 INFO:journalctl@ceph.mon.a.smithi002.stdout: return future.result(timeout) 2024-09-17T00:22:15.981 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/lib64/python3.9/concurrent/futures/_base.py", line 446, in result 2024-09-17T00:22:15.981 INFO:journalctl@ceph.mon.a.smithi002.stdout: return self.__get_result() 2024-09-17T00:22:15.981 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/lib64/python3.9/concurrent/futures/_base.py", line 391, in __get_result 2024-09-17T00:22:15.981 INFO:journalctl@ceph.mon.a.smithi002.stdout: raise self._exception 2024-09-17T00:22:15.981 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1370, in _create_daemon 2024-09-17T00:22:15.981 INFO:journalctl@ceph.mon.a.smithi002.stdout: out, err, code = await self._run_cephadm( 2024-09-17T00:22:15.982 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1713, in _run_cephadm 2024-09-17T00:22:15.982 INFO:journalctl@ceph.mon.a.smithi002.stdout: raise OrchestratorError( 2024-09-17T00:22:15.982 INFO:journalctl@ceph.mon.a.smithi002.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi002.vmouem ... 2024-09-17T00:22:15.982 INFO:journalctl@ceph.mon.a.smithi002.stdout: Creating ceph-iscsi config... 2024-09-17T00:22:15.982 INFO:journalctl@ceph.mon.a.smithi002.stdout: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/iscsi-gateway.cfg 2024-09-17T00:22:15.982 INFO:journalctl@ceph.mon.a.smithi002.stdout: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/tcmu-runner-entrypoint.sh 2024-09-17T00:22:15.982 INFO:journalctl@ceph.mon.a.smithi002.stdout: Non-zero exit code 1 from systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem 2024-09-17T00:22:15.982 INFO:journalctl@ceph.mon.a.smithi002.stdout: systemctl: stderr Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:22:15.982 INFO:journalctl@ceph.mon.a.smithi002.stdout: systemctl: stderr See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:22:15.982 INFO:journalctl@ceph.mon.a.smithi002.stdout: Traceback (most recent call last): 2024-09-17T00:22:15.982 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-17T00:22:15.982 INFO:journalctl@ceph.mon.a.smithi002.stdout: return _run_code(code, main_globals, None, 2024-09-17T00:22:15.982 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-17T00:22:15.982 INFO:journalctl@ceph.mon.a.smithi002.stdout: exec(code, run_globals) 2024-09-17T00:22:15.982 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5579, in 2024-09-17T00:22:15.983 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5567, in main 2024-09-17T00:22:15.983 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3049, in command_deploy_from 2024-09-17T00:22:15.983 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3084, in _common_deploy 2024-09-17T00:22:15.983 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3104, in _deploy_daemon_container 2024-09-17T00:22:15.983 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 1124, in deploy_daemon 2024-09-17T00:22:15.983 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-17T00:22:15.983 INFO:journalctl@ceph.mon.a.smithi002.stdout: RuntimeError: Failed command: systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem: Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:22:15.983 INFO:journalctl@ceph.mon.a.smithi002.stdout: See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:22:15.983 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:15 smithi002 ceph-mon[23737]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-17T00:22:15.983 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:15 smithi002 ceph-mon[23737]: Deploying daemon prometheus.a on smithi099 2024-09-17T00:22:15.983 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:15 smithi002 ceph-mon[28203]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi002.vmouem ... 2024-09-17T00:22:15.984 INFO:journalctl@ceph.mon.c.smithi002.stdout: Creating ceph-iscsi config... 2024-09-17T00:22:15.984 INFO:journalctl@ceph.mon.c.smithi002.stdout: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/iscsi-gateway.cfg 2024-09-17T00:22:15.984 INFO:journalctl@ceph.mon.c.smithi002.stdout: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/tcmu-runner-entrypoint.sh 2024-09-17T00:22:15.984 INFO:journalctl@ceph.mon.c.smithi002.stdout: Non-zero exit code 1 from systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem 2024-09-17T00:22:15.984 INFO:journalctl@ceph.mon.c.smithi002.stdout: systemctl: stderr Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:22:15.984 INFO:journalctl@ceph.mon.c.smithi002.stdout: systemctl: stderr See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:22:15.984 INFO:journalctl@ceph.mon.c.smithi002.stdout: Traceback (most recent call last): 2024-09-17T00:22:15.984 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-17T00:22:15.984 INFO:journalctl@ceph.mon.c.smithi002.stdout: return _run_code(code, main_globals, None, 2024-09-17T00:22:15.984 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-17T00:22:15.984 INFO:journalctl@ceph.mon.c.smithi002.stdout: exec(code, run_globals) 2024-09-17T00:22:15.984 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5579, in 2024-09-17T00:22:15.985 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5567, in main 2024-09-17T00:22:15.985 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3049, in command_deploy_from 2024-09-17T00:22:15.985 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3084, in _common_deploy 2024-09-17T00:22:15.985 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3104, in _deploy_daemon_container 2024-09-17T00:22:15.985 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 1124, in deploy_daemon 2024-09-17T00:22:15.985 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-17T00:22:15.985 INFO:journalctl@ceph.mon.c.smithi002.stdout: RuntimeError: Failed command: systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem: Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:22:15.985 INFO:journalctl@ceph.mon.c.smithi002.stdout: See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:22:15.985 INFO:journalctl@ceph.mon.c.smithi002.stdout: Traceback (most recent call last): 2024-09-17T00:22:15.985 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1134, in _check_daemons 2024-09-17T00:22:15.985 INFO:journalctl@ceph.mon.c.smithi002.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-09-17T00:22:15.985 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2473, in _daemon_action 2024-09-17T00:22:15.985 INFO:journalctl@ceph.mon.c.smithi002.stdout: return self.wait_async( 2024-09-17T00:22:15.985 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 796, in wait_async 2024-09-17T00:22:15.986 INFO:journalctl@ceph.mon.c.smithi002.stdout: return self.event_loop.get_result(coro, timeout) 2024-09-17T00:22:15.986 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-09-17T00:22:15.986 INFO:journalctl@ceph.mon.c.smithi002.stdout: return future.result(timeout) 2024-09-17T00:22:15.986 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/lib64/python3.9/concurrent/futures/_base.py", line 446, in result 2024-09-17T00:22:15.986 INFO:journalctl@ceph.mon.c.smithi002.stdout: return self.__get_result() 2024-09-17T00:22:15.986 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/lib64/python3.9/concurrent/futures/_base.py", line 391, in __get_result 2024-09-17T00:22:15.986 INFO:journalctl@ceph.mon.c.smithi002.stdout: raise self._exception 2024-09-17T00:22:15.986 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1370, in _create_daemon 2024-09-17T00:22:15.986 INFO:journalctl@ceph.mon.c.smithi002.stdout: out, err, code = await self._run_cephadm( 2024-09-17T00:22:15.986 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1713, in _run_cephadm 2024-09-17T00:22:15.986 INFO:journalctl@ceph.mon.c.smithi002.stdout: raise OrchestratorError( 2024-09-17T00:22:15.986 INFO:journalctl@ceph.mon.c.smithi002.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi002.vmouem ... 2024-09-17T00:22:15.986 INFO:journalctl@ceph.mon.c.smithi002.stdout: Creating ceph-iscsi config... 2024-09-17T00:22:15.986 INFO:journalctl@ceph.mon.c.smithi002.stdout: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/iscsi-gateway.cfg 2024-09-17T00:22:15.987 INFO:journalctl@ceph.mon.c.smithi002.stdout: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/tcmu-runner-entrypoint.sh 2024-09-17T00:22:15.987 INFO:journalctl@ceph.mon.c.smithi002.stdout: Non-zero exit code 1 from systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem 2024-09-17T00:22:15.987 INFO:journalctl@ceph.mon.c.smithi002.stdout: systemctl: stderr Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:22:15.987 INFO:journalctl@ceph.mon.c.smithi002.stdout: systemctl: stderr See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:22:15.987 INFO:journalctl@ceph.mon.c.smithi002.stdout: Traceback (most recent call last): 2024-09-17T00:22:15.987 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-17T00:22:15.987 INFO:journalctl@ceph.mon.c.smithi002.stdout: return _run_code(code, main_globals, None, 2024-09-17T00:22:15.987 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-17T00:22:15.987 INFO:journalctl@ceph.mon.c.smithi002.stdout: exec(code, run_globals) 2024-09-17T00:22:15.987 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5579, in 2024-09-17T00:22:15.987 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5567, in main 2024-09-17T00:22:15.987 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3049, in command_deploy_from 2024-09-17T00:22:15.987 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3084, in _common_deploy 2024-09-17T00:22:15.988 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3104, in _deploy_daemon_container 2024-09-17T00:22:15.988 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 1124, in deploy_daemon 2024-09-17T00:22:15.988 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-17T00:22:15.988 INFO:journalctl@ceph.mon.c.smithi002.stdout: RuntimeError: Failed command: systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem: Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:22:15.988 INFO:journalctl@ceph.mon.c.smithi002.stdout: See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:22:15.988 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:15 smithi002 ceph-mon[28203]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-17T00:22:15.988 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:15 smithi002 ceph-mon[28203]: Deploying daemon prometheus.a on smithi099 2024-09-17T00:22:15.988 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:15 smithi002 ceph-mon[23737]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:22:15.988 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:15 smithi002 ceph-mon[23737]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T00:22:15.989 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:15 smithi002 ceph-mon[28203]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:22:15.989 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:15 smithi002 ceph-mon[28203]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T00:22:17.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:17 smithi099 ceph-mon[24594]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-17T00:22:17.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:17 smithi002 ceph-mon[28203]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-17T00:22:17.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:17 smithi002 ceph-mon[23737]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-17T00:22:19.479 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:22:19 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[23964]: ::ffff:172.21.15.99 - - [17/Sep/2024:00:22:19] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-09-17T00:22:19.966 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:19 smithi099 ceph-mon[24594]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-09-17T00:22:19.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:19 smithi002 ceph-mon[28203]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-09-17T00:22:19.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:19 smithi002 ceph-mon[23737]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-09-17T00:22:20.979 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Sep 17 00:22:20 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-alertmanager-a[58000]: ts=2024-09-17T00:22:20.491Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.002762154s 2024-09-17T00:22:21.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:21 smithi099 ceph-mon[24594]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-09-17T00:22:21.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:21 smithi002 ceph-mon[28203]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-09-17T00:22:21.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:21 smithi002 ceph-mon[23737]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-09-17T00:22:22.717 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:22:22 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[41464]: ts=2024-09-17T00:22:22.314Z caller=manager.go:609 level=warn component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.99:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.2:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-17T00:22:23.966 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:23 smithi099 ceph-mon[24594]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-09-17T00:22:23.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:23 smithi002 ceph-mon[28203]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-09-17T00:22:23.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:23 smithi002 ceph-mon[23737]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-09-17T00:22:25.217 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:22:24 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: ::ffff:172.21.15.99 - - [17/Sep/2024:00:22:24] "GET /metrics HTTP/1.1" 200 38131 "" "Prometheus/2.33.4" 2024-09-17T00:22:25.217 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:22:24 smithi099 systemd[1]: Stopping Ceph prometheus.a for 3aad27a2-7489-11ef-bceb-c7b262605968... 2024-09-17T00:22:25.218 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:22:24 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[41464]: ts=2024-09-17T00:22:24.959Z caller=main.go:775 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-09-17T00:22:25.218 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:22:24 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[41464]: ts=2024-09-17T00:22:24.959Z caller=main.go:798 level=info msg="Stopping scrape discovery manager..." 2024-09-17T00:22:25.218 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:22:24 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[41464]: ts=2024-09-17T00:22:24.959Z caller=main.go:812 level=info msg="Stopping notify discovery manager..." 2024-09-17T00:22:25.218 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:22:24 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[41464]: ts=2024-09-17T00:22:24.959Z caller=main.go:834 level=info msg="Stopping scrape manager..." 2024-09-17T00:22:25.218 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:22:24 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[41464]: ts=2024-09-17T00:22:24.959Z caller=main.go:794 level=info msg="Scrape discovery manager stopped" 2024-09-17T00:22:25.218 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:22:24 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[41464]: ts=2024-09-17T00:22:24.959Z caller=main.go:808 level=info msg="Notify discovery manager stopped" 2024-09-17T00:22:25.218 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:22:24 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[41464]: ts=2024-09-17T00:22:24.959Z caller=manager.go:945 level=info component="rule manager" msg="Stopping rule manager..." 2024-09-17T00:22:25.218 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:22:24 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[41464]: ts=2024-09-17T00:22:24.959Z caller=main.go:828 level=info msg="Scrape manager stopped" 2024-09-17T00:22:25.218 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:22:24 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[41464]: ts=2024-09-17T00:22:24.959Z caller=manager.go:955 level=info component="rule manager" msg="Rule manager stopped" 2024-09-17T00:22:25.218 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:22:24 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[41464]: ts=2024-09-17T00:22:24.959Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..." 2024-09-17T00:22:25.218 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:22:24 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[41464]: ts=2024-09-17T00:22:24.959Z caller=main.go:1054 level=info msg="Notifier manager stopped" 2024-09-17T00:22:25.218 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:22:24 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[41464]: ts=2024-09-17T00:22:24.959Z caller=main.go:1066 level=info msg="See you next time!" 2024-09-17T00:22:25.218 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:22:24 smithi099 podman[49212]: 2024-09-17 00:22:24.969338155 +0000 UTC m=+0.104904974 container died a1529c2c7a1cafe7d2add603f09411f6a07e99ebd29fd45d99dc5adbc76ca919 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-17T00:22:25.623 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:22:25 smithi099 podman[49212]: 2024-09-17 00:22:25.316989861 +0000 UTC m=+0.452556679 container cleanup a1529c2c7a1cafe7d2add603f09411f6a07e99ebd29fd45d99dc5adbc76ca919 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-17T00:22:25.623 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:22:25 smithi099 bash[49212]: ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a 2024-09-17T00:22:25.966 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:25 smithi099 ceph-mon[24594]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-09-17T00:22:25.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:25 smithi099 ceph-mon[24594]: from='client.? 172.21.15.2:0/2039019892' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-17T00:22:25.967 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:22:25 smithi099 podman[49224]: 2024-09-17 00:22:25.626757131 +0000 UTC m=+0.659790572 container remove a1529c2c7a1cafe7d2add603f09411f6a07e99ebd29fd45d99dc5adbc76ca919 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-17T00:22:25.967 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:22:25 smithi099 systemd[1]: ceph-3aad27a2-7489-11ef-bceb-c7b262605968@prometheus.a.service: Deactivated successfully. 2024-09-17T00:22:25.967 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:22:25 smithi099 systemd[1]: Stopped Ceph prometheus.a for 3aad27a2-7489-11ef-bceb-c7b262605968. 2024-09-17T00:22:25.967 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:22:25 smithi099 systemd[1]: ceph-3aad27a2-7489-11ef-bceb-c7b262605968@prometheus.a.service: Consumed 1.882s CPU time. 2024-09-17T00:22:25.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:25 smithi002 ceph-mon[28203]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-09-17T00:22:25.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:25 smithi002 ceph-mon[28203]: from='client.? 172.21.15.2:0/2039019892' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-17T00:22:25.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:25 smithi002 ceph-mon[23737]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-09-17T00:22:25.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:25 smithi002 ceph-mon[23737]: from='client.? 172.21.15.2:0/2039019892' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-17T00:22:26.338 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:22:25 smithi099 systemd[1]: Starting Ceph prometheus.a for 3aad27a2-7489-11ef-bceb-c7b262605968... 2024-09-17T00:22:26.338 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:22:26 smithi099 podman[49354]: 2024-09-17 00:22:26.100572003 +0000 UTC m=+0.016793104 image pull a07b618ecd1dce142bce4c52f0e80982eaf1f14265a2415c2d35978ccaa0a464 quay.io/prometheus/prometheus:v2.43.0 2024-09-17T00:22:26.338 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:22:26 smithi099 podman[49354]: 2024-09-17 00:22:26.219795287 +0000 UTC m=+0.136016372 container create 1d14ff0e495ea18aeb3efae630ad2af8b255b51b51b50169ab3e0077fac846d2 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-17T00:22:26.712 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:22:26 smithi099 podman[49354]: 2024-09-17 00:22:26.438971286 +0000 UTC m=+0.355192371 container init 1d14ff0e495ea18aeb3efae630ad2af8b255b51b51b50169ab3e0077fac846d2 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-17T00:22:26.712 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:22:26 smithi099 podman[49354]: 2024-09-17 00:22:26.441676068 +0000 UTC m=+0.357897147 container start 1d14ff0e495ea18aeb3efae630ad2af8b255b51b51b50169ab3e0077fac846d2 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-17T00:22:26.712 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:22:26 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[49368]: ts=2024-09-17T00:22:26.466Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-09-17T00:22:26.712 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:22:26 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[49368]: ts=2024-09-17T00:22:26.466Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-09-17T00:22:26.712 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:22:26 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[49368]: ts=2024-09-17T00:22:26.466Z caller=main.go:570 level=info host_details="(Linux 5.14.0-508.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024 x86_64 smithi099 (none))" 2024-09-17T00:22:26.712 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:22:26 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[49368]: ts=2024-09-17T00:22:26.466Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-09-17T00:22:26.712 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:22:26 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[49368]: ts=2024-09-17T00:22:26.467Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-09-17T00:22:26.713 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:22:26 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[49368]: ts=2024-09-17T00:22:26.468Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-09-17T00:22:26.713 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:22:26 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[49368]: ts=2024-09-17T00:22:26.468Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-09-17T00:22:26.713 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:22:26 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[49368]: ts=2024-09-17T00:22:26.470Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-09-17T00:22:26.713 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:22:26 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[49368]: ts=2024-09-17T00:22:26.470Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-09-17T00:22:26.713 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:22:26 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[49368]: ts=2024-09-17T00:22:26.471Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-09-17T00:22:26.713 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:22:26 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[49368]: ts=2024-09-17T00:22:26.471Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=2.069µs 2024-09-17T00:22:26.713 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:22:26 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[49368]: ts=2024-09-17T00:22:26.471Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-09-17T00:22:26.713 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:22:26 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[49368]: ts=2024-09-17T00:22:26.479Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=2 2024-09-17T00:22:26.713 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:22:26 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[49368]: ts=2024-09-17T00:22:26.490Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=2 2024-09-17T00:22:26.713 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:22:26 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[49368]: ts=2024-09-17T00:22:26.491Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=2 maxSegment=2 2024-09-17T00:22:26.713 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:22:26 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[49368]: ts=2024-09-17T00:22:26.491Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=19.422µs wal_replay_duration=19.539597ms wbl_replay_duration=112ns total_replay_duration=19.579858ms 2024-09-17T00:22:26.713 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:22:26 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[49368]: ts=2024-09-17T00:22:26.493Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-09-17T00:22:26.714 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:22:26 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[49368]: ts=2024-09-17T00:22:26.493Z caller=main.go:1029 level=info msg="TSDB started" 2024-09-17T00:22:26.714 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:22:26 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[49368]: ts=2024-09-17T00:22:26.493Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-09-17T00:22:26.714 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:22:26 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[49368]: ts=2024-09-17T00:22:26.514Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=21.100702ms db_storage=791ns remote_storage=1.105µs web_handler=424ns query_engine=645ns scrape=692.493µs scrape_sd=100.497µs notify=15.44µs notify_sd=18.341µs rules=19.873962ms tracing=6.829µs 2024-09-17T00:22:26.714 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:22:26 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[49368]: ts=2024-09-17T00:22:26.514Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-09-17T00:22:26.714 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:22:26 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[49368]: ts=2024-09-17T00:22:26.514Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-09-17T00:22:26.714 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:22:26 smithi099 bash[49354]: 1d14ff0e495ea18aeb3efae630ad2af8b255b51b51b50169ab3e0077fac846d2 2024-09-17T00:22:26.714 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:22:26 smithi099 systemd[1]: Started Ceph prometheus.a for 3aad27a2-7489-11ef-bceb-c7b262605968. 2024-09-17T00:22:26.966 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:26 smithi099 ceph-mon[24594]: from='client.? 172.21.15.2:0/3251067612' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/2517426025"}]: dispatch 2024-09-17T00:22:26.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:26 smithi099 ceph-mon[24594]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:22:26.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:26 smithi099 ceph-mon[24594]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:22:26.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:26 smithi002 ceph-mon[28203]: from='client.? 172.21.15.2:0/3251067612' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/2517426025"}]: dispatch 2024-09-17T00:22:26.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:26 smithi002 ceph-mon[28203]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:22:26.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:26 smithi002 ceph-mon[28203]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:22:26.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:26 smithi002 ceph-mon[23737]: from='client.? 172.21.15.2:0/3251067612' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/2517426025"}]: dispatch 2024-09-17T00:22:26.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:26 smithi002 ceph-mon[23737]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:22:26.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:26 smithi002 ceph-mon[23737]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:22:27.467 INFO:journalctl@ceph.node-exporter.b.smithi099.stdout:Sep 17 00:22:27 smithi099 systemd[1]: Stopping Ceph node-exporter.b for 3aad27a2-7489-11ef-bceb-c7b262605968... 2024-09-17T00:22:27.966 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:27 smithi099 ceph-mon[24594]: Reconfiguring node-exporter.b (dependencies changed)... 2024-09-17T00:22:27.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:27 smithi099 ceph-mon[24594]: Deploying daemon node-exporter.b on smithi099 2024-09-17T00:22:27.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:27 smithi099 ceph-mon[24594]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-09-17T00:22:27.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:27 smithi099 ceph-mon[24594]: from='client.? 172.21.15.2:0/3251067612' entity='client.iscsi.foo.smithi002.vmouem' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/2517426025"}]': finished 2024-09-17T00:22:27.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:27 smithi099 ceph-mon[24594]: osdmap e98: 8 total, 8 up, 8 in 2024-09-17T00:22:27.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:27 smithi099 ceph-mon[24594]: from='client.? 172.21.15.2:0/3831597803' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/3887064165"}]: dispatch 2024-09-17T00:22:27.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:27 smithi099 ceph-mon[24594]: from='client.? ' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/3887064165"}]: dispatch 2024-09-17T00:22:27.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:27 smithi002 ceph-mon[28203]: Reconfiguring node-exporter.b (dependencies changed)... 2024-09-17T00:22:27.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:27 smithi002 ceph-mon[28203]: Deploying daemon node-exporter.b on smithi099 2024-09-17T00:22:27.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:27 smithi002 ceph-mon[28203]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-09-17T00:22:27.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:27 smithi002 ceph-mon[28203]: from='client.? 172.21.15.2:0/3251067612' entity='client.iscsi.foo.smithi002.vmouem' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/2517426025"}]': finished 2024-09-17T00:22:27.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:27 smithi002 ceph-mon[28203]: osdmap e98: 8 total, 8 up, 8 in 2024-09-17T00:22:27.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:27 smithi002 ceph-mon[28203]: from='client.? 172.21.15.2:0/3831597803' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/3887064165"}]: dispatch 2024-09-17T00:22:27.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:27 smithi002 ceph-mon[28203]: from='client.? ' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/3887064165"}]: dispatch 2024-09-17T00:22:27.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:27 smithi002 ceph-mon[23737]: Reconfiguring node-exporter.b (dependencies changed)... 2024-09-17T00:22:27.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:27 smithi002 ceph-mon[23737]: Deploying daemon node-exporter.b on smithi099 2024-09-17T00:22:27.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:27 smithi002 ceph-mon[23737]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-09-17T00:22:27.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:27 smithi002 ceph-mon[23737]: from='client.? 172.21.15.2:0/3251067612' entity='client.iscsi.foo.smithi002.vmouem' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/2517426025"}]': finished 2024-09-17T00:22:27.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:27 smithi002 ceph-mon[23737]: osdmap e98: 8 total, 8 up, 8 in 2024-09-17T00:22:27.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:27 smithi002 ceph-mon[23737]: from='client.? 172.21.15.2:0/3831597803' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/3887064165"}]: dispatch 2024-09-17T00:22:27.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:27 smithi002 ceph-mon[23737]: from='client.? ' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/3887064165"}]: dispatch 2024-09-17T00:22:28.716 INFO:journalctl@ceph.node-exporter.b.smithi099.stdout:Sep 17 00:22:28 smithi099 podman[49533]: 2024-09-17 00:22:28.471900364 +0000 UTC m=+1.113535610 container died 352ae73fe2a7d92d222a4b36681d326961a29b84383478e8bc975f417e124933 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-17T00:22:28.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:28 smithi002 ceph-mon[23737]: from='client.? ' entity='client.iscsi.foo.smithi002.vmouem' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/3887064165"}]': finished 2024-09-17T00:22:28.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:28 smithi002 ceph-mon[23737]: osdmap e99: 8 total, 8 up, 8 in 2024-09-17T00:22:28.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:28 smithi002 ceph-mon[23737]: from='client.? 172.21.15.2:0/3713082562' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/4027284037"}]: dispatch 2024-09-17T00:22:28.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:28 smithi002 ceph-mon[23737]: from='client.? ' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/4027284037"}]: dispatch 2024-09-17T00:22:28.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:28 smithi002 ceph-mon[23737]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 127 B/s rd, 0 op/s 2024-09-17T00:22:28.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:28 smithi002 ceph-mon[28203]: from='client.? ' entity='client.iscsi.foo.smithi002.vmouem' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/3887064165"}]': finished 2024-09-17T00:22:28.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:28 smithi002 ceph-mon[28203]: osdmap e99: 8 total, 8 up, 8 in 2024-09-17T00:22:28.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:28 smithi002 ceph-mon[28203]: from='client.? 172.21.15.2:0/3713082562' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/4027284037"}]: dispatch 2024-09-17T00:22:28.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:28 smithi002 ceph-mon[28203]: from='client.? ' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/4027284037"}]: dispatch 2024-09-17T00:22:28.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:28 smithi002 ceph-mon[28203]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 127 B/s rd, 0 op/s 2024-09-17T00:22:29.016 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:28 smithi099 ceph-mon[24594]: from='client.? ' entity='client.iscsi.foo.smithi002.vmouem' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/3887064165"}]': finished 2024-09-17T00:22:29.016 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:28 smithi099 ceph-mon[24594]: osdmap e99: 8 total, 8 up, 8 in 2024-09-17T00:22:29.016 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:28 smithi099 ceph-mon[24594]: from='client.? 172.21.15.2:0/3713082562' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/4027284037"}]: dispatch 2024-09-17T00:22:29.016 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:28 smithi099 ceph-mon[24594]: from='client.? ' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/4027284037"}]: dispatch 2024-09-17T00:22:29.016 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:28 smithi099 ceph-mon[24594]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 127 B/s rd, 0 op/s 2024-09-17T00:22:29.309 INFO:journalctl@ceph.node-exporter.b.smithi099.stdout:Sep 17 00:22:29 smithi099 podman[49533]: 2024-09-17 00:22:29.019848731 +0000 UTC m=+1.661483976 container cleanup 352ae73fe2a7d92d222a4b36681d326961a29b84383478e8bc975f417e124933 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-17T00:22:29.309 INFO:journalctl@ceph.node-exporter.b.smithi099.stdout:Sep 17 00:22:29 smithi099 bash[49533]: ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-b 2024-09-17T00:22:29.649 INFO:journalctl@ceph.node-exporter.b.smithi099.stdout:Sep 17 00:22:29 smithi099 podman[49555]: 2024-09-17 00:22:29.313002774 +0000 UTC m=+0.844985921 container remove 352ae73fe2a7d92d222a4b36681d326961a29b84383478e8bc975f417e124933 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-17T00:22:29.649 INFO:journalctl@ceph.node-exporter.b.smithi099.stdout:Sep 17 00:22:29 smithi099 systemd[1]: ceph-3aad27a2-7489-11ef-bceb-c7b262605968@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-09-17T00:22:29.649 INFO:journalctl@ceph.node-exporter.b.smithi099.stdout:Sep 17 00:22:29 smithi099 systemd[1]: ceph-3aad27a2-7489-11ef-bceb-c7b262605968@node-exporter.b.service: Failed with result 'exit-code'. 2024-09-17T00:22:29.649 INFO:journalctl@ceph.node-exporter.b.smithi099.stdout:Sep 17 00:22:29 smithi099 systemd[1]: Stopped Ceph node-exporter.b for 3aad27a2-7489-11ef-bceb-c7b262605968. 2024-09-17T00:22:29.649 INFO:journalctl@ceph.node-exporter.b.smithi099.stdout:Sep 17 00:22:29 smithi099 systemd[1]: ceph-3aad27a2-7489-11ef-bceb-c7b262605968@node-exporter.b.service: Consumed 3.280s CPU time. 2024-09-17T00:22:29.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:29 smithi099 ceph-mon[24594]: from='client.? ' entity='client.iscsi.foo.smithi002.vmouem' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/4027284037"}]': finished 2024-09-17T00:22:29.968 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:29 smithi099 ceph-mon[24594]: osdmap e100: 8 total, 8 up, 8 in 2024-09-17T00:22:29.968 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:29 smithi099 ceph-mon[24594]: from='client.? 172.21.15.2:0/857218123' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/2967708425"}]: dispatch 2024-09-17T00:22:29.968 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:29 smithi099 ceph-mon[24594]: from='client.? ' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/2967708425"}]: dispatch 2024-09-17T00:22:29.968 INFO:journalctl@ceph.node-exporter.b.smithi099.stdout:Sep 17 00:22:29 smithi099 systemd[1]: Starting Ceph node-exporter.b for 3aad27a2-7489-11ef-bceb-c7b262605968... 2024-09-17T00:22:29.969 INFO:journalctl@ceph.node-exporter.b.smithi099.stdout:Sep 17 00:22:29 smithi099 bash[49684]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-09-17T00:22:30.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:29 smithi002 ceph-mon[23737]: from='client.? ' entity='client.iscsi.foo.smithi002.vmouem' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/4027284037"}]': finished 2024-09-17T00:22:30.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:29 smithi002 ceph-mon[23737]: osdmap e100: 8 total, 8 up, 8 in 2024-09-17T00:22:30.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:29 smithi002 ceph-mon[23737]: from='client.? 172.21.15.2:0/857218123' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/2967708425"}]: dispatch 2024-09-17T00:22:30.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:29 smithi002 ceph-mon[23737]: from='client.? ' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/2967708425"}]: dispatch 2024-09-17T00:22:30.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:29 smithi002 ceph-mon[28203]: from='client.? ' entity='client.iscsi.foo.smithi002.vmouem' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/4027284037"}]': finished 2024-09-17T00:22:30.230 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:29 smithi002 ceph-mon[28203]: osdmap e100: 8 total, 8 up, 8 in 2024-09-17T00:22:30.230 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:29 smithi002 ceph-mon[28203]: from='client.? 172.21.15.2:0/857218123' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/2967708425"}]: dispatch 2024-09-17T00:22:30.230 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:29 smithi002 ceph-mon[28203]: from='client.? ' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/2967708425"}]: dispatch 2024-09-17T00:22:30.717 INFO:journalctl@ceph.node-exporter.b.smithi099.stdout:Sep 17 00:22:30 smithi099 bash[49684]: Getting image source signatures 2024-09-17T00:22:30.717 INFO:journalctl@ceph.node-exporter.b.smithi099.stdout:Sep 17 00:22:30 smithi099 bash[49684]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-09-17T00:22:30.717 INFO:journalctl@ceph.node-exporter.b.smithi099.stdout:Sep 17 00:22:30 smithi099 bash[49684]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-09-17T00:22:30.717 INFO:journalctl@ceph.node-exporter.b.smithi099.stdout:Sep 17 00:22:30 smithi099 bash[49684]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-09-17T00:22:31.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:30 smithi099 ceph-mon[24594]: from='client.? ' entity='client.iscsi.foo.smithi002.vmouem' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/2967708425"}]': finished 2024-09-17T00:22:31.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:30 smithi099 ceph-mon[24594]: osdmap e101: 8 total, 8 up, 8 in 2024-09-17T00:22:31.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:30 smithi099 ceph-mon[24594]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T00:22:31.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:30 smithi099 ceph-mon[24594]: from='client.? 172.21.15.2:0/19369326' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6801/1824194566"}]: dispatch 2024-09-17T00:22:31.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:30 smithi099 ceph-mon[24594]: from='client.? ' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6801/1824194566"}]: dispatch 2024-09-17T00:22:31.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:30 smithi099 ceph-mon[24594]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:22:31.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:30 smithi002 ceph-mon[28203]: from='client.? ' entity='client.iscsi.foo.smithi002.vmouem' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/2967708425"}]': finished 2024-09-17T00:22:31.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:30 smithi002 ceph-mon[28203]: osdmap e101: 8 total, 8 up, 8 in 2024-09-17T00:22:31.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:30 smithi002 ceph-mon[28203]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T00:22:31.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:30 smithi002 ceph-mon[28203]: from='client.? 172.21.15.2:0/19369326' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6801/1824194566"}]: dispatch 2024-09-17T00:22:31.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:30 smithi002 ceph-mon[28203]: from='client.? ' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6801/1824194566"}]: dispatch 2024-09-17T00:22:31.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:30 smithi002 ceph-mon[28203]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:22:31.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:30 smithi002 ceph-mon[23737]: from='client.? ' entity='client.iscsi.foo.smithi002.vmouem' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/2967708425"}]': finished 2024-09-17T00:22:31.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:30 smithi002 ceph-mon[23737]: osdmap e101: 8 total, 8 up, 8 in 2024-09-17T00:22:31.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:30 smithi002 ceph-mon[23737]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T00:22:31.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:30 smithi002 ceph-mon[23737]: from='client.? 172.21.15.2:0/19369326' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6801/1824194566"}]: dispatch 2024-09-17T00:22:31.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:30 smithi002 ceph-mon[23737]: from='client.? ' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6801/1824194566"}]: dispatch 2024-09-17T00:22:31.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:30 smithi002 ceph-mon[23737]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:22:32.017 INFO:journalctl@ceph.node-exporter.b.smithi099.stdout:Sep 17 00:22:31 smithi099 bash[49684]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-09-17T00:22:32.018 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:31 smithi099 ceph-mon[24594]: from='client.? ' entity='client.iscsi.foo.smithi002.vmouem' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6801/1824194566"}]': finished 2024-09-17T00:22:32.018 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:31 smithi099 ceph-mon[24594]: osdmap e102: 8 total, 8 up, 8 in 2024-09-17T00:22:32.018 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:31 smithi099 ceph-mon[24594]: from='client.? 172.21.15.2:0/1891826895' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6800/1824194566"}]: dispatch 2024-09-17T00:22:32.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:31 smithi002 ceph-mon[28203]: from='client.? ' entity='client.iscsi.foo.smithi002.vmouem' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6801/1824194566"}]': finished 2024-09-17T00:22:32.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:31 smithi002 ceph-mon[28203]: osdmap e102: 8 total, 8 up, 8 in 2024-09-17T00:22:32.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:31 smithi002 ceph-mon[28203]: from='client.? 172.21.15.2:0/1891826895' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6800/1824194566"}]: dispatch 2024-09-17T00:22:32.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:31 smithi002 ceph-mon[23737]: from='client.? ' entity='client.iscsi.foo.smithi002.vmouem' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6801/1824194566"}]': finished 2024-09-17T00:22:32.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:31 smithi002 ceph-mon[23737]: osdmap e102: 8 total, 8 up, 8 in 2024-09-17T00:22:32.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:31 smithi002 ceph-mon[23737]: from='client.? 172.21.15.2:0/1891826895' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6800/1824194566"}]: dispatch 2024-09-17T00:22:32.369 INFO:journalctl@ceph.node-exporter.b.smithi099.stdout:Sep 17 00:22:32 smithi099 bash[49684]: Writing manifest to image destination 2024-09-17T00:22:32.656 INFO:journalctl@ceph.node-exporter.b.smithi099.stdout:Sep 17 00:22:32 smithi099 podman[49684]: 2024-09-17 00:22:32.276087486 +0000 UTC m=+2.524807327 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-09-17T00:22:32.656 INFO:journalctl@ceph.node-exporter.b.smithi099.stdout:Sep 17 00:22:32 smithi099 podman[49684]: 2024-09-17 00:22:32.408003489 +0000 UTC m=+2.656723331 container create 01fcd35dca8a80afb348bd4c20910055034ea1182396da895bde6f280f96c996 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-17T00:22:32.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:32 smithi099 ceph-mon[24594]: from='client.? 172.21.15.2:0/1891826895' entity='client.iscsi.foo.smithi002.vmouem' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6800/1824194566"}]': finished 2024-09-17T00:22:32.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:32 smithi099 ceph-mon[24594]: osdmap e103: 8 total, 8 up, 8 in 2024-09-17T00:22:32.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:32 smithi099 ceph-mon[24594]: from='client.? 172.21.15.2:0/1008355768' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/3574422273"}]: dispatch 2024-09-17T00:22:32.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:32 smithi099 ceph-mon[24594]: from='client.? ' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/3574422273"}]: dispatch 2024-09-17T00:22:32.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:32 smithi099 ceph-mon[24594]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:22:32.967 INFO:journalctl@ceph.node-exporter.b.smithi099.stdout:Sep 17 00:22:32 smithi099 podman[49684]: 2024-09-17 00:22:32.659926461 +0000 UTC m=+2.908646310 container init 01fcd35dca8a80afb348bd4c20910055034ea1182396da895bde6f280f96c996 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-17T00:22:32.968 INFO:journalctl@ceph.node-exporter.b.smithi099.stdout:Sep 17 00:22:32 smithi099 podman[49684]: 2024-09-17 00:22:32.662828352 +0000 UTC m=+2.911548194 container start 01fcd35dca8a80afb348bd4c20910055034ea1182396da895bde6f280f96c996 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-17T00:22:32.968 INFO:journalctl@ceph.node-exporter.b.smithi099.stdout:Sep 17 00:22:32 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-b[49759]: ts=2024-09-17T00:22:32.666Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-09-17T00:22:32.968 INFO:journalctl@ceph.node-exporter.b.smithi099.stdout:Sep 17 00:22:32 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-b[49759]: ts=2024-09-17T00:22:32.666Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-09-17T00:22:32.968 INFO:journalctl@ceph.node-exporter.b.smithi099.stdout:Sep 17 00:22:32 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-b[49759]: ts=2024-09-17T00:22:32.667Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-09-17T00:22:32.968 INFO:journalctl@ceph.node-exporter.b.smithi099.stdout:Sep 17 00:22:32 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-b[49759]: ts=2024-09-17T00:22:32.667Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-09-17T00:22:32.968 INFO:journalctl@ceph.node-exporter.b.smithi099.stdout:Sep 17 00:22:32 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-b[49759]: ts=2024-09-17T00:22:32.667Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-09-17T00:22:32.968 INFO:journalctl@ceph.node-exporter.b.smithi099.stdout:Sep 17 00:22:32 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-b[49759]: ts=2024-09-17T00:22:32.667Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-09-17T00:22:32.968 INFO:journalctl@ceph.node-exporter.b.smithi099.stdout:Sep 17 00:22:32 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-b[49759]: ts=2024-09-17T00:22:32.668Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-09-17T00:22:32.968 INFO:journalctl@ceph.node-exporter.b.smithi099.stdout:Sep 17 00:22:32 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-b[49759]: ts=2024-09-17T00:22:32.668Z caller=node_exporter.go:117 level=info collector=arp 2024-09-17T00:22:32.968 INFO:journalctl@ceph.node-exporter.b.smithi099.stdout:Sep 17 00:22:32 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-b[49759]: ts=2024-09-17T00:22:32.668Z caller=node_exporter.go:117 level=info collector=bcache 2024-09-17T00:22:32.968 INFO:journalctl@ceph.node-exporter.b.smithi099.stdout:Sep 17 00:22:32 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-b[49759]: ts=2024-09-17T00:22:32.668Z caller=node_exporter.go:117 level=info collector=bonding 2024-09-17T00:22:32.969 INFO:journalctl@ceph.node-exporter.b.smithi099.stdout:Sep 17 00:22:32 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-b[49759]: ts=2024-09-17T00:22:32.668Z caller=node_exporter.go:117 level=info collector=btrfs 2024-09-17T00:22:32.969 INFO:journalctl@ceph.node-exporter.b.smithi099.stdout:Sep 17 00:22:32 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-b[49759]: ts=2024-09-17T00:22:32.668Z caller=node_exporter.go:117 level=info collector=conntrack 2024-09-17T00:22:32.969 INFO:journalctl@ceph.node-exporter.b.smithi099.stdout:Sep 17 00:22:32 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-b[49759]: ts=2024-09-17T00:22:32.668Z caller=node_exporter.go:117 level=info collector=cpu 2024-09-17T00:22:32.969 INFO:journalctl@ceph.node-exporter.b.smithi099.stdout:Sep 17 00:22:32 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-b[49759]: ts=2024-09-17T00:22:32.668Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-09-17T00:22:32.969 INFO:journalctl@ceph.node-exporter.b.smithi099.stdout:Sep 17 00:22:32 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-b[49759]: ts=2024-09-17T00:22:32.668Z caller=node_exporter.go:117 level=info collector=diskstats 2024-09-17T00:22:32.969 INFO:journalctl@ceph.node-exporter.b.smithi099.stdout:Sep 17 00:22:32 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-b[49759]: ts=2024-09-17T00:22:32.668Z caller=node_exporter.go:117 level=info collector=dmi 2024-09-17T00:22:32.969 INFO:journalctl@ceph.node-exporter.b.smithi099.stdout:Sep 17 00:22:32 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-b[49759]: ts=2024-09-17T00:22:32.668Z caller=node_exporter.go:117 level=info collector=edac 2024-09-17T00:22:32.970 INFO:journalctl@ceph.node-exporter.b.smithi099.stdout:Sep 17 00:22:32 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-b[49759]: ts=2024-09-17T00:22:32.668Z caller=node_exporter.go:117 level=info collector=entropy 2024-09-17T00:22:32.970 INFO:journalctl@ceph.node-exporter.b.smithi099.stdout:Sep 17 00:22:32 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-b[49759]: ts=2024-09-17T00:22:32.668Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-09-17T00:22:32.970 INFO:journalctl@ceph.node-exporter.b.smithi099.stdout:Sep 17 00:22:32 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-b[49759]: ts=2024-09-17T00:22:32.668Z caller=node_exporter.go:117 level=info collector=filefd 2024-09-17T00:22:32.970 INFO:journalctl@ceph.node-exporter.b.smithi099.stdout:Sep 17 00:22:32 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-b[49759]: ts=2024-09-17T00:22:32.668Z caller=node_exporter.go:117 level=info collector=filesystem 2024-09-17T00:22:32.970 INFO:journalctl@ceph.node-exporter.b.smithi099.stdout:Sep 17 00:22:32 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-b[49759]: ts=2024-09-17T00:22:32.668Z caller=node_exporter.go:117 level=info collector=hwmon 2024-09-17T00:22:32.970 INFO:journalctl@ceph.node-exporter.b.smithi099.stdout:Sep 17 00:22:32 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-b[49759]: ts=2024-09-17T00:22:32.668Z caller=node_exporter.go:117 level=info collector=infiniband 2024-09-17T00:22:32.970 INFO:journalctl@ceph.node-exporter.b.smithi099.stdout:Sep 17 00:22:32 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-b[49759]: ts=2024-09-17T00:22:32.668Z caller=node_exporter.go:117 level=info collector=ipvs 2024-09-17T00:22:32.970 INFO:journalctl@ceph.node-exporter.b.smithi099.stdout:Sep 17 00:22:32 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-b[49759]: ts=2024-09-17T00:22:32.668Z caller=node_exporter.go:117 level=info collector=loadavg 2024-09-17T00:22:32.970 INFO:journalctl@ceph.node-exporter.b.smithi099.stdout:Sep 17 00:22:32 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-b[49759]: ts=2024-09-17T00:22:32.668Z caller=node_exporter.go:117 level=info collector=mdadm 2024-09-17T00:22:32.970 INFO:journalctl@ceph.node-exporter.b.smithi099.stdout:Sep 17 00:22:32 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-b[49759]: ts=2024-09-17T00:22:32.668Z caller=node_exporter.go:117 level=info collector=meminfo 2024-09-17T00:22:32.970 INFO:journalctl@ceph.node-exporter.b.smithi099.stdout:Sep 17 00:22:32 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-b[49759]: ts=2024-09-17T00:22:32.668Z caller=node_exporter.go:117 level=info collector=netclass 2024-09-17T00:22:32.970 INFO:journalctl@ceph.node-exporter.b.smithi099.stdout:Sep 17 00:22:32 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-b[49759]: ts=2024-09-17T00:22:32.668Z caller=node_exporter.go:117 level=info collector=netdev 2024-09-17T00:22:32.970 INFO:journalctl@ceph.node-exporter.b.smithi099.stdout:Sep 17 00:22:32 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-b[49759]: ts=2024-09-17T00:22:32.668Z caller=node_exporter.go:117 level=info collector=netstat 2024-09-17T00:22:32.970 INFO:journalctl@ceph.node-exporter.b.smithi099.stdout:Sep 17 00:22:32 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-b[49759]: ts=2024-09-17T00:22:32.668Z caller=node_exporter.go:117 level=info collector=nfs 2024-09-17T00:22:32.971 INFO:journalctl@ceph.node-exporter.b.smithi099.stdout:Sep 17 00:22:32 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-b[49759]: ts=2024-09-17T00:22:32.668Z caller=node_exporter.go:117 level=info collector=nfsd 2024-09-17T00:22:32.971 INFO:journalctl@ceph.node-exporter.b.smithi099.stdout:Sep 17 00:22:32 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-b[49759]: ts=2024-09-17T00:22:32.668Z caller=node_exporter.go:117 level=info collector=nvme 2024-09-17T00:22:32.971 INFO:journalctl@ceph.node-exporter.b.smithi099.stdout:Sep 17 00:22:32 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-b[49759]: ts=2024-09-17T00:22:32.668Z caller=node_exporter.go:117 level=info collector=os 2024-09-17T00:22:32.971 INFO:journalctl@ceph.node-exporter.b.smithi099.stdout:Sep 17 00:22:32 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-b[49759]: ts=2024-09-17T00:22:32.668Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-09-17T00:22:32.971 INFO:journalctl@ceph.node-exporter.b.smithi099.stdout:Sep 17 00:22:32 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-b[49759]: ts=2024-09-17T00:22:32.668Z caller=node_exporter.go:117 level=info collector=pressure 2024-09-17T00:22:32.971 INFO:journalctl@ceph.node-exporter.b.smithi099.stdout:Sep 17 00:22:32 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-b[49759]: ts=2024-09-17T00:22:32.668Z caller=node_exporter.go:117 level=info collector=rapl 2024-09-17T00:22:32.971 INFO:journalctl@ceph.node-exporter.b.smithi099.stdout:Sep 17 00:22:32 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-b[49759]: ts=2024-09-17T00:22:32.668Z caller=node_exporter.go:117 level=info collector=schedstat 2024-09-17T00:22:32.971 INFO:journalctl@ceph.node-exporter.b.smithi099.stdout:Sep 17 00:22:32 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-b[49759]: ts=2024-09-17T00:22:32.668Z caller=node_exporter.go:117 level=info collector=selinux 2024-09-17T00:22:32.971 INFO:journalctl@ceph.node-exporter.b.smithi099.stdout:Sep 17 00:22:32 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-b[49759]: ts=2024-09-17T00:22:32.668Z caller=node_exporter.go:117 level=info collector=sockstat 2024-09-17T00:22:32.971 INFO:journalctl@ceph.node-exporter.b.smithi099.stdout:Sep 17 00:22:32 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-b[49759]: ts=2024-09-17T00:22:32.668Z caller=node_exporter.go:117 level=info collector=softnet 2024-09-17T00:22:32.971 INFO:journalctl@ceph.node-exporter.b.smithi099.stdout:Sep 17 00:22:32 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-b[49759]: ts=2024-09-17T00:22:32.668Z caller=node_exporter.go:117 level=info collector=stat 2024-09-17T00:22:32.971 INFO:journalctl@ceph.node-exporter.b.smithi099.stdout:Sep 17 00:22:32 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-b[49759]: ts=2024-09-17T00:22:32.668Z caller=node_exporter.go:117 level=info collector=tapestats 2024-09-17T00:22:32.971 INFO:journalctl@ceph.node-exporter.b.smithi099.stdout:Sep 17 00:22:32 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-b[49759]: ts=2024-09-17T00:22:32.668Z caller=node_exporter.go:117 level=info collector=textfile 2024-09-17T00:22:32.971 INFO:journalctl@ceph.node-exporter.b.smithi099.stdout:Sep 17 00:22:32 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-b[49759]: ts=2024-09-17T00:22:32.668Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-09-17T00:22:32.972 INFO:journalctl@ceph.node-exporter.b.smithi099.stdout:Sep 17 00:22:32 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-b[49759]: ts=2024-09-17T00:22:32.668Z caller=node_exporter.go:117 level=info collector=time 2024-09-17T00:22:32.972 INFO:journalctl@ceph.node-exporter.b.smithi099.stdout:Sep 17 00:22:32 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-b[49759]: ts=2024-09-17T00:22:32.668Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-09-17T00:22:32.972 INFO:journalctl@ceph.node-exporter.b.smithi099.stdout:Sep 17 00:22:32 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-b[49759]: ts=2024-09-17T00:22:32.668Z caller=node_exporter.go:117 level=info collector=uname 2024-09-17T00:22:32.972 INFO:journalctl@ceph.node-exporter.b.smithi099.stdout:Sep 17 00:22:32 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-b[49759]: ts=2024-09-17T00:22:32.668Z caller=node_exporter.go:117 level=info collector=vmstat 2024-09-17T00:22:32.972 INFO:journalctl@ceph.node-exporter.b.smithi099.stdout:Sep 17 00:22:32 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-b[49759]: ts=2024-09-17T00:22:32.668Z caller=node_exporter.go:117 level=info collector=xfs 2024-09-17T00:22:32.972 INFO:journalctl@ceph.node-exporter.b.smithi099.stdout:Sep 17 00:22:32 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-b[49759]: ts=2024-09-17T00:22:32.668Z caller=node_exporter.go:117 level=info collector=zfs 2024-09-17T00:22:32.972 INFO:journalctl@ceph.node-exporter.b.smithi099.stdout:Sep 17 00:22:32 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-b[49759]: ts=2024-09-17T00:22:32.669Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-09-17T00:22:32.972 INFO:journalctl@ceph.node-exporter.b.smithi099.stdout:Sep 17 00:22:32 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-b[49759]: ts=2024-09-17T00:22:32.669Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-09-17T00:22:32.972 INFO:journalctl@ceph.node-exporter.b.smithi099.stdout:Sep 17 00:22:32 smithi099 bash[49684]: 01fcd35dca8a80afb348bd4c20910055034ea1182396da895bde6f280f96c996 2024-09-17T00:22:32.972 INFO:journalctl@ceph.node-exporter.b.smithi099.stdout:Sep 17 00:22:32 smithi099 systemd[1]: Started Ceph node-exporter.b for 3aad27a2-7489-11ef-bceb-c7b262605968. 2024-09-17T00:22:33.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:32 smithi002 ceph-mon[28203]: from='client.? 172.21.15.2:0/1891826895' entity='client.iscsi.foo.smithi002.vmouem' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6800/1824194566"}]': finished 2024-09-17T00:22:33.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:32 smithi002 ceph-mon[28203]: osdmap e103: 8 total, 8 up, 8 in 2024-09-17T00:22:33.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:32 smithi002 ceph-mon[28203]: from='client.? 172.21.15.2:0/1008355768' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/3574422273"}]: dispatch 2024-09-17T00:22:33.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:32 smithi002 ceph-mon[28203]: from='client.? ' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/3574422273"}]: dispatch 2024-09-17T00:22:33.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:32 smithi002 ceph-mon[28203]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:22:33.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:32 smithi002 ceph-mon[23737]: from='client.? 172.21.15.2:0/1891826895' entity='client.iscsi.foo.smithi002.vmouem' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6800/1824194566"}]': finished 2024-09-17T00:22:33.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:32 smithi002 ceph-mon[23737]: osdmap e103: 8 total, 8 up, 8 in 2024-09-17T00:22:33.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:32 smithi002 ceph-mon[23737]: from='client.? 172.21.15.2:0/1008355768' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/3574422273"}]: dispatch 2024-09-17T00:22:33.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:32 smithi002 ceph-mon[23737]: from='client.? ' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/3574422273"}]: dispatch 2024-09-17T00:22:33.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:32 smithi002 ceph-mon[23737]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:22:33.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:33 smithi099 ceph-mon[24594]: from='client.? ' entity='client.iscsi.foo.smithi002.vmouem' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/3574422273"}]': finished 2024-09-17T00:22:33.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:33 smithi099 ceph-mon[24594]: osdmap e104: 8 total, 8 up, 8 in 2024-09-17T00:22:33.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:33 smithi099 ceph-mon[24594]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:22:33.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:33 smithi099 ceph-mon[24594]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:22:33.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:33 smithi099 ceph-mon[24594]: Reconfiguring grafana.a (dependencies changed)... 2024-09-17T00:22:33.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:33 smithi099 ceph-mon[24594]: Regenerating cephadm self-signed grafana TLS certificates 2024-09-17T00:22:33.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:33 smithi099 ceph-mon[24594]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:22:33.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:33 smithi099 ceph-mon[24594]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:22:33.968 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:33 smithi099 ceph-mon[24594]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-17T00:22:33.968 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:33 smithi099 ceph-mon[24594]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-17T00:22:33.968 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:33 smithi099 ceph-mon[24594]: Reconfiguring daemon grafana.a on smithi099 2024-09-17T00:22:34.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:33 smithi002 ceph-mon[23737]: from='client.? ' entity='client.iscsi.foo.smithi002.vmouem' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/3574422273"}]': finished 2024-09-17T00:22:34.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:33 smithi002 ceph-mon[23737]: osdmap e104: 8 total, 8 up, 8 in 2024-09-17T00:22:34.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:33 smithi002 ceph-mon[23737]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:22:34.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:33 smithi002 ceph-mon[23737]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:22:34.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:33 smithi002 ceph-mon[23737]: Reconfiguring grafana.a (dependencies changed)... 2024-09-17T00:22:34.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:33 smithi002 ceph-mon[23737]: Regenerating cephadm self-signed grafana TLS certificates 2024-09-17T00:22:34.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:33 smithi002 ceph-mon[23737]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:22:34.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:33 smithi002 ceph-mon[23737]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:22:34.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:33 smithi002 ceph-mon[23737]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-17T00:22:34.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:33 smithi002 ceph-mon[23737]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-17T00:22:34.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:33 smithi002 ceph-mon[23737]: Reconfiguring daemon grafana.a on smithi099 2024-09-17T00:22:34.230 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:33 smithi002 ceph-mon[28203]: from='client.? ' entity='client.iscsi.foo.smithi002.vmouem' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/3574422273"}]': finished 2024-09-17T00:22:34.230 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:33 smithi002 ceph-mon[28203]: osdmap e104: 8 total, 8 up, 8 in 2024-09-17T00:22:34.230 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:33 smithi002 ceph-mon[28203]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:22:34.230 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:33 smithi002 ceph-mon[28203]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:22:34.230 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:33 smithi002 ceph-mon[28203]: Reconfiguring grafana.a (dependencies changed)... 2024-09-17T00:22:34.230 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:33 smithi002 ceph-mon[28203]: Regenerating cephadm self-signed grafana TLS certificates 2024-09-17T00:22:34.231 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:33 smithi002 ceph-mon[28203]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:22:34.231 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:33 smithi002 ceph-mon[28203]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:22:34.231 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:33 smithi002 ceph-mon[28203]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-17T00:22:34.231 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:33 smithi002 ceph-mon[28203]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-17T00:22:34.231 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:33 smithi002 ceph-mon[28203]: Reconfiguring daemon grafana.a on smithi099 2024-09-17T00:22:34.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:34 smithi099 ceph-mon[24594]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-17T00:22:35.210 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:34 smithi002 ceph-mon[28203]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-17T00:22:35.210 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:34 smithi002 ceph-mon[23737]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-17T00:22:35.759 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:22:35 smithi099 systemd[1]: Stopping Ceph grafana.a for 3aad27a2-7489-11ef-bceb-c7b262605968... 2024-09-17T00:22:36.105 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:35 smithi099 ceph-mon[24594]: from='client.15243 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:22:36.105 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:22:35 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[40687]: t=2024-09-17T00:22:35+0000 lvl=info msg="Shutdown started" logger=server reason="System signal: terminated" 2024-09-17T00:22:36.106 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:22:35 smithi099 podman[49935]: 2024-09-17 00:22:35.824301416 +0000 UTC m=+0.140314756 container died 1547cd1718321080b01ab2a29b7a75ac7922c3f7b9a739466a69137091fd539f (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-type=git, build-date=2022-03-28T10:36:18.413762, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., io.buildah.version=1.24.2, distribution-scope=public, io.openshift.expose-services=, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.component=ubi8-container, architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, description=Ceph Grafana Container, io.k8s.display-name=Red Hat Universal Base Image 8, name=ubi8, release=236.1648460182, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.openshift.tags=base rhel8, version=8.5, maintainer=Paul Cuzner ) 2024-09-17T00:22:36.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:35 smithi002 ceph-mon[23737]: from='client.15243 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:22:36.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:35 smithi002 ceph-mon[28203]: from='client.15243 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:22:36.467 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:22:36 smithi099 podman[49935]: 2024-09-17 00:22:36.188926755 +0000 UTC m=+0.504940085 container cleanup 1547cd1718321080b01ab2a29b7a75ac7922c3f7b9a739466a69137091fd539f (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a, vendor=Red Hat, Inc., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=, maintainer=Paul Cuzner , version=8.5, vcs-type=git, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.openshift.tags=base rhel8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, description=Ceph Grafana Container, distribution-scope=public, name=ubi8, architecture=x86_64, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=236.1648460182, io.buildah.version=1.24.2, io.k8s.display-name=Red Hat Universal Base Image 8, build-date=2022-03-28T10:36:18.413762, com.redhat.component=ubi8-container) 2024-09-17T00:22:36.467 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:22:36 smithi099 bash[49935]: ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a 2024-09-17T00:22:36.467 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:22:36 smithi099 bash[49962]: Error: no container with name or ID "ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana.a" found: no such container 2024-09-17T00:22:36.738 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:22:36 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: ::ffff:172.21.15.99 - - [17/Sep/2024:00:22:36] "GET /metrics HTTP/1.1" 200 38132 "" "Prometheus/2.43.0" 2024-09-17T00:22:36.738 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:22:36 smithi099 podman[49947]: 2024-09-17 00:22:36.481878006 +0000 UTC m=+0.656693566 container remove 1547cd1718321080b01ab2a29b7a75ac7922c3f7b9a739466a69137091fd539f (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a, description=Ceph Grafana Container, name=ubi8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=ubi8-container, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, architecture=x86_64, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=Red Hat Universal Base Image 8, build-date=2022-03-28T10:36:18.413762, release=236.1648460182, vcs-type=git, io.buildah.version=1.24.2, io.openshift.tags=base rhel8, maintainer=Paul Cuzner , io.openshift.expose-services=, version=8.5, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, summary=Grafana Container configured for Ceph mgr/dashboard integration, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, vendor=Red Hat, Inc.) 2024-09-17T00:22:36.738 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:22:36 smithi099 systemd[1]: ceph-3aad27a2-7489-11ef-bceb-c7b262605968@grafana.a.service: Deactivated successfully. 2024-09-17T00:22:36.738 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:22:36 smithi099 systemd[1]: Stopped Ceph grafana.a for 3aad27a2-7489-11ef-bceb-c7b262605968. 2024-09-17T00:22:36.738 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:22:36 smithi099 systemd[1]: ceph-3aad27a2-7489-11ef-bceb-c7b262605968@grafana.a.service: Consumed 2.503s CPU time. 2024-09-17T00:22:36.738 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:22:36 smithi099 systemd[1]: Starting Ceph grafana.a for 3aad27a2-7489-11ef-bceb-c7b262605968... 2024-09-17T00:22:36.738 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:22:36 smithi099 podman[50026]: 2024-09-17 00:22:36.615957596 +0000 UTC m=+0.015793234 image pull dad864ee21e98e69f4029d1e417aa085001566be0d322fbc75bc6f29b0050c01 quay.io/ceph/ceph-grafana:8.3.5 2024-09-17T00:22:36.738 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:22:36 smithi099 podman[50026]: 2024-09-17 00:22:36.741217568 +0000 UTC m=+0.141053237 container create bafb824a3c0802109f37ce34f4f387b35333f335f235abda3a6afb6e8e067e00 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a, architecture=x86_64, com.redhat.component=ubi8-container, io.openshift.expose-services=, vcs-type=git, io.k8s.display-name=Red Hat Universal Base Image 8, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, description=Ceph Grafana Container, distribution-scope=public, io.openshift.tags=base rhel8, maintainer=Paul Cuzner , com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, build-date=2022-03-28T10:36:18.413762, name=ubi8, version=8.5, io.buildah.version=1.24.2, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, summary=Grafana Container configured for Ceph mgr/dashboard integration, release=236.1648460182, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56) 2024-09-17T00:22:36.990 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:36 smithi099 ceph-mon[24594]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-17T00:22:37.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:36 smithi002 ceph-mon[23737]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-17T00:22:37.230 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:36 smithi002 ceph-mon[28203]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-17T00:22:37.334 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:22:37 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: [17/Sep/2024:00:22:37] ENGINE Bus STOPPING 2024-09-17T00:22:37.335 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:22:37 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: [17/Sep/2024:00:22:37] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-17T00:22:37.335 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:22:37 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: [17/Sep/2024:00:22:37] ENGINE Bus STOPPED 2024-09-17T00:22:37.335 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:22:37 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: [17/Sep/2024:00:22:37] ENGINE Bus STARTING 2024-09-17T00:22:37.335 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:22:37 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: [17/Sep/2024:00:22:37] ENGINE Serving on http://:::9283 2024-09-17T00:22:37.335 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:22:37 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: [17/Sep/2024:00:22:37] ENGINE Bus STARTED 2024-09-17T00:22:37.335 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:22:36 smithi099 podman[50026]: 2024-09-17 00:22:36.993930092 +0000 UTC m=+0.393765720 container init bafb824a3c0802109f37ce34f4f387b35333f335f235abda3a6afb6e8e067e00 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a, name=ubi8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., architecture=x86_64, io.buildah.version=1.24.2, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, com.redhat.component=ubi8-container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, version=8.5, summary=Grafana Container configured for Ceph mgr/dashboard integration, vendor=Red Hat, Inc., vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, description=Ceph Grafana Container, distribution-scope=public, io.openshift.tags=base rhel8, maintainer=Paul Cuzner , com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, release=236.1648460182, vcs-type=git, io.k8s.display-name=Red Hat Universal Base Image 8, build-date=2022-03-28T10:36:18.413762, io.openshift.expose-services=) 2024-09-17T00:22:37.335 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:22:36 smithi099 podman[50026]: 2024-09-17 00:22:36.997279718 +0000 UTC m=+0.397115344 container start bafb824a3c0802109f37ce34f4f387b35333f335f235abda3a6afb6e8e067e00 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a, maintainer=Paul Cuzner , io.buildah.version=1.24.2, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-type=git, io.openshift.expose-services=, vendor=Red Hat, Inc., name=ubi8, io.openshift.tags=base rhel8, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, description=Ceph Grafana Container, distribution-scope=public, io.k8s.display-name=Red Hat Universal Base Image 8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, build-date=2022-03-28T10:36:18.413762, summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.component=ubi8-container, version=8.5, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=236.1648460182, architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182) 2024-09-17T00:22:37.336 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:22:37 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[50042]: t=2024-09-17T00:22:37+0000 lvl=info msg="The state of unified alerting is still not defined. The decision will be made during as we run the database migrations" logger=settings 2024-09-17T00:22:37.336 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:22:37 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[50042]: t=2024-09-17T00:22:37+0000 lvl=warn msg="falling back to legacy setting of 'min_interval_seconds'; please use the configuration option in the `unified_alerting` section if Grafana 8 alerts are enabled." logger=settings 2024-09-17T00:22:37.336 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:22:37 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[50042]: t=2024-09-17T00:22:37+0000 lvl=info msg="Config loaded from" logger=settings file=/usr/share/grafana/conf/defaults.ini 2024-09-17T00:22:37.336 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:22:37 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[50042]: t=2024-09-17T00:22:37+0000 lvl=info msg="Config loaded from" logger=settings file=/etc/grafana/grafana.ini 2024-09-17T00:22:37.336 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:22:37 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[50042]: t=2024-09-17T00:22:37+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_DATA=/var/lib/grafana" 2024-09-17T00:22:37.336 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:22:37 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[50042]: t=2024-09-17T00:22:37+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_LOGS=/var/log/grafana" 2024-09-17T00:22:37.336 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:22:37 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[50042]: t=2024-09-17T00:22:37+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-09-17T00:22:37.336 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:22:37 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[50042]: t=2024-09-17T00:22:37+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-09-17T00:22:37.336 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:22:37 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[50042]: t=2024-09-17T00:22:37+0000 lvl=info msg="Path Home" logger=settings path=/usr/share/grafana 2024-09-17T00:22:37.336 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:22:37 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[50042]: t=2024-09-17T00:22:37+0000 lvl=info msg="Path Data" logger=settings path=/var/lib/grafana 2024-09-17T00:22:37.336 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:22:37 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[50042]: t=2024-09-17T00:22:37+0000 lvl=info msg="Path Logs" logger=settings path=/var/log/grafana 2024-09-17T00:22:37.336 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:22:37 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[50042]: t=2024-09-17T00:22:37+0000 lvl=info msg="Path Plugins" logger=settings path=/var/lib/grafana/plugins 2024-09-17T00:22:37.336 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:22:37 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[50042]: t=2024-09-17T00:22:37+0000 lvl=info msg="Path Provisioning" logger=settings path=/etc/grafana/provisioning 2024-09-17T00:22:37.337 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:22:37 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[50042]: t=2024-09-17T00:22:37+0000 lvl=info msg="App mode production" logger=settings 2024-09-17T00:22:37.337 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:22:37 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[50042]: t=2024-09-17T00:22:37+0000 lvl=info msg="Connecting to DB" logger=sqlstore dbtype=sqlite3 2024-09-17T00:22:37.337 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:22:37 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[50042]: t=2024-09-17T00:22:37+0000 lvl=warn msg="SQLite database file has broader permissions than it should" logger=sqlstore path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-09-17T00:22:37.337 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:22:37 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[50042]: t=2024-09-17T00:22:37+0000 lvl=info msg="Starting DB migrations" logger=migrator 2024-09-17T00:22:37.337 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:22:37 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[50042]: t=2024-09-17T00:22:37+0000 lvl=info msg="migrations completed" logger=migrator performed=0 skipped=377 duration=510.277µs 2024-09-17T00:22:37.337 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:22:37 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[50042]: t=2024-09-17T00:22:37+0000 lvl=info msg="Created default organization" logger=sqlstore 2024-09-17T00:22:37.337 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:22:37 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[50042]: t=2024-09-17T00:22:37+0000 lvl=info msg="Initialising plugins" logger=plugin.manager 2024-09-17T00:22:37.337 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:22:37 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[50042]: t=2024-09-17T00:22:37+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=input 2024-09-17T00:22:37.337 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:22:37 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[50042]: t=2024-09-17T00:22:37+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=grafana-piechart-panel 2024-09-17T00:22:37.337 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:22:37 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[50042]: t=2024-09-17T00:22:37+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=vonage-status-panel 2024-09-17T00:22:37.337 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:22:37 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[50042]: t=2024-09-17T00:22:37+0000 lvl=info msg="Live Push Gateway initialization" logger=live.push_http 2024-09-17T00:22:37.337 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:22:37 smithi099 bash[50026]: bafb824a3c0802109f37ce34f4f387b35333f335f235abda3a6afb6e8e067e00 2024-09-17T00:22:37.337 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:22:37 smithi099 systemd[1]: Started Ceph grafana.a for 3aad27a2-7489-11ef-bceb-c7b262605968. 2024-09-17T00:22:37.338 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:22:37 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[50042]: t=2024-09-17T00:22:37+0000 lvl=info msg="deleted datasource based on configuration" logger=provisioning.datasources name=Dashboard1 2024-09-17T00:22:37.338 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:22:37 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[50042]: t=2024-09-17T00:22:37+0000 lvl=info msg="inserting datasource from configuration " logger=provisioning.datasources name=Dashboard1 uid=P43CA22E17D0F9596 2024-09-17T00:22:37.338 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:22:37 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[50042]: t=2024-09-17T00:22:37+0000 lvl=info msg="inserting datasource from configuration " logger=provisioning.datasources name=Loki uid=P8E80F9AEF21F6940 2024-09-17T00:22:37.338 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:22:37 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[50042]: t=2024-09-17T00:22:37+0000 lvl=info msg="warming cache for startup" logger=ngalert 2024-09-17T00:22:37.338 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:22:37 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[50042]: t=2024-09-17T00:22:37+0000 lvl=info msg="HTTP Server Listen" logger=http.server address=[::]:3000 protocol=https subUrl= socket= 2024-09-17T00:22:37.338 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:22:37 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[50042]: t=2024-09-17T00:22:37+0000 lvl=info msg="starting MultiOrg Alertmanager" logger=ngalert.multiorg.alertmanager 2024-09-17T00:22:37.717 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:22:37 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: [17/Sep/2024:00:22:37] ENGINE Bus STOPPING 2024-09-17T00:22:37.831 INFO:teuthology.orchestra.run.smithi002.stdout:true 2024-09-17T00:22:38.134 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:22:37 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: [17/Sep/2024:00:22:37] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-17T00:22:38.134 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:22:37 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: [17/Sep/2024:00:22:37] ENGINE Bus STOPPED 2024-09-17T00:22:38.134 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:22:37 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: [17/Sep/2024:00:22:37] ENGINE Bus STARTING 2024-09-17T00:22:38.134 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:22:37 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: [17/Sep/2024:00:22:37] ENGINE Serving on http://:::9283 2024-09-17T00:22:38.134 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:22:37 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: [17/Sep/2024:00:22:37] ENGINE Bus STARTED 2024-09-17T00:22:38.134 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:22:37 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: [17/Sep/2024:00:22:37] ENGINE Bus STOPPING 2024-09-17T00:22:38.449 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:38 smithi099 ceph-mon[24594]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:22:38.450 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:38 smithi099 ceph-mon[24594]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:22:38.450 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:38 smithi099 ceph-mon[24594]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-17T00:22:38.450 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:38 smithi099 ceph-mon[24594]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-17T00:22:38.450 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:38 smithi099 ceph-mon[24594]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi002.front.sepia.ceph.com:9093"}]: dispatch 2024-09-17T00:22:38.450 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:38 smithi099 ceph-mon[24594]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi002.front.sepia.ceph.com:9093"}]: dispatch 2024-09-17T00:22:38.450 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:38 smithi099 ceph-mon[24594]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:22:38.450 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:38 smithi099 ceph-mon[24594]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-17T00:22:38.450 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:38 smithi099 ceph-mon[24594]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-17T00:22:38.450 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:38 smithi099 ceph-mon[24594]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi099.front.sepia.ceph.com:9095"}]: dispatch 2024-09-17T00:22:38.450 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:38 smithi099 ceph-mon[24594]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi099.front.sepia.ceph.com:9095"}]: dispatch 2024-09-17T00:22:38.450 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:38 smithi099 ceph-mon[24594]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:22:38.450 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:38 smithi099 ceph-mon[24594]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-17T00:22:38.451 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:38 smithi099 ceph-mon[24594]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-17T00:22:38.451 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:38 smithi099 ceph-mon[24594]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi099.front.sepia.ceph.com:3000"}]: dispatch 2024-09-17T00:22:38.451 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:38 smithi099 ceph-mon[24594]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi099.front.sepia.ceph.com:3000"}]: dispatch 2024-09-17T00:22:38.451 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:38 smithi099 ceph-mon[24594]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:22:38.451 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:38 smithi099 ceph-mon[24594]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T00:22:38.451 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:38 smithi099 ceph-mon[24594]: Upgrade: Need to upgrade myself (mgr.x) 2024-09-17T00:22:38.451 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:38 smithi099 ceph-mon[24594]: Upgrade: Need to upgrade myself (mgr.x) 2024-09-17T00:22:38.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:38 smithi002 ceph-mon[23737]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:22:38.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:38 smithi002 ceph-mon[23737]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:22:38.480 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:38 smithi002 ceph-mon[23737]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-17T00:22:38.480 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:38 smithi002 ceph-mon[23737]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-17T00:22:38.480 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:38 smithi002 ceph-mon[23737]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi002.front.sepia.ceph.com:9093"}]: dispatch 2024-09-17T00:22:38.480 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:38 smithi002 ceph-mon[23737]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi002.front.sepia.ceph.com:9093"}]: dispatch 2024-09-17T00:22:38.480 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:38 smithi002 ceph-mon[23737]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:22:38.480 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:38 smithi002 ceph-mon[23737]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-17T00:22:38.480 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:38 smithi002 ceph-mon[23737]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-17T00:22:38.480 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:38 smithi002 ceph-mon[23737]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi099.front.sepia.ceph.com:9095"}]: dispatch 2024-09-17T00:22:38.480 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:38 smithi002 ceph-mon[23737]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi099.front.sepia.ceph.com:9095"}]: dispatch 2024-09-17T00:22:38.480 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:38 smithi002 ceph-mon[23737]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:22:38.480 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:38 smithi002 ceph-mon[23737]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-17T00:22:38.480 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:38 smithi002 ceph-mon[23737]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-17T00:22:38.481 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:38 smithi002 ceph-mon[23737]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi099.front.sepia.ceph.com:3000"}]: dispatch 2024-09-17T00:22:38.481 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:38 smithi002 ceph-mon[23737]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi099.front.sepia.ceph.com:3000"}]: dispatch 2024-09-17T00:22:38.481 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:38 smithi002 ceph-mon[23737]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:22:38.481 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:38 smithi002 ceph-mon[23737]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T00:22:38.481 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:38 smithi002 ceph-mon[23737]: Upgrade: Need to upgrade myself (mgr.x) 2024-09-17T00:22:38.481 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:38 smithi002 ceph-mon[23737]: Upgrade: Need to upgrade myself (mgr.x) 2024-09-17T00:22:38.481 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:38 smithi002 ceph-mon[28203]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:22:38.481 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:38 smithi002 ceph-mon[28203]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:22:38.481 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:38 smithi002 ceph-mon[28203]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-17T00:22:38.481 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:38 smithi002 ceph-mon[28203]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-17T00:22:38.481 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:38 smithi002 ceph-mon[28203]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi002.front.sepia.ceph.com:9093"}]: dispatch 2024-09-17T00:22:38.481 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:38 smithi002 ceph-mon[28203]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi002.front.sepia.ceph.com:9093"}]: dispatch 2024-09-17T00:22:38.482 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:38 smithi002 ceph-mon[28203]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:22:38.482 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:38 smithi002 ceph-mon[28203]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-17T00:22:38.482 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:38 smithi002 ceph-mon[28203]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-17T00:22:38.482 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:38 smithi002 ceph-mon[28203]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi099.front.sepia.ceph.com:9095"}]: dispatch 2024-09-17T00:22:38.482 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:38 smithi002 ceph-mon[28203]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi099.front.sepia.ceph.com:9095"}]: dispatch 2024-09-17T00:22:38.482 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:38 smithi002 ceph-mon[28203]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:22:38.482 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:38 smithi002 ceph-mon[28203]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-17T00:22:38.482 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:38 smithi002 ceph-mon[28203]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-17T00:22:38.482 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:38 smithi002 ceph-mon[28203]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi099.front.sepia.ceph.com:3000"}]: dispatch 2024-09-17T00:22:38.482 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:38 smithi002 ceph-mon[28203]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi099.front.sepia.ceph.com:3000"}]: dispatch 2024-09-17T00:22:38.482 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:38 smithi002 ceph-mon[28203]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:22:38.482 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:38 smithi002 ceph-mon[28203]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T00:22:38.482 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:38 smithi002 ceph-mon[28203]: Upgrade: Need to upgrade myself (mgr.x) 2024-09-17T00:22:38.483 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:38 smithi002 ceph-mon[28203]: Upgrade: Need to upgrade myself (mgr.x) 2024-09-17T00:22:38.558 INFO:teuthology.orchestra.run.smithi002.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-17T00:22:38.558 INFO:teuthology.orchestra.run.smithi002.stdout:alertmanager.a smithi002 *:9093,9094 starting - - - - 2024-09-17T00:22:38.558 INFO:teuthology.orchestra.run.smithi002.stdout:grafana.a smithi099 *:3000 starting - - - - 2024-09-17T00:22:38.559 INFO:teuthology.orchestra.run.smithi002.stdout:iscsi.foo.smithi002.vmouem smithi002 running (4m) 47s ago 4m 52.1M - 3.5 e1d6a67b021e 6b47041ec00f 2024-09-17T00:22:38.559 INFO:teuthology.orchestra.run.smithi002.stdout:mgr.x smithi099 *:8443,9283 running (63s) 47s ago 9m 481M - 19.1.1-330-gc2ddf972 ebb0615149de a8e1e8964c84 2024-09-17T00:22:38.559 INFO:teuthology.orchestra.run.smithi002.stdout:mgr.y smithi002 *:9283 running (11m) 47s ago 11m 173M - 17.2.0 e1d6a67b021e d32b04d294d3 2024-09-17T00:22:38.559 INFO:teuthology.orchestra.run.smithi002.stdout:mon.a smithi002 running (11m) 47s ago 11m 54.2M 2048M 17.2.0 e1d6a67b021e a68823df625f 2024-09-17T00:22:38.559 INFO:teuthology.orchestra.run.smithi002.stdout:mon.b smithi099 running (10m) 47s ago 10m 44.2M 2048M 17.2.0 e1d6a67b021e c6d2af715187 2024-09-17T00:22:38.559 INFO:teuthology.orchestra.run.smithi002.stdout:mon.c smithi002 running (10m) 47s ago 10m 41.1M 2048M 17.2.0 e1d6a67b021e 77b3d2e14899 2024-09-17T00:22:38.559 INFO:teuthology.orchestra.run.smithi002.stdout:node-exporter.a smithi002 *:9100 starting - - - - 2024-09-17T00:22:38.559 INFO:teuthology.orchestra.run.smithi002.stdout:node-exporter.b smithi099 *:9100 starting - - - - 2024-09-17T00:22:38.559 INFO:teuthology.orchestra.run.smithi002.stdout:osd.0 smithi002 running (9m) 47s ago 9m 52.4M 2737M 17.2.0 e1d6a67b021e a0ad0fb7aa34 2024-09-17T00:22:38.559 INFO:teuthology.orchestra.run.smithi002.stdout:osd.1 smithi002 running (9m) 47s ago 9m 54.5M 2737M 17.2.0 e1d6a67b021e 7cb62c311c7d 2024-09-17T00:22:38.559 INFO:teuthology.orchestra.run.smithi002.stdout:osd.2 smithi002 running (8m) 47s ago 8m 50.7M 2737M 17.2.0 e1d6a67b021e eafddacb955e 2024-09-17T00:22:38.559 INFO:teuthology.orchestra.run.smithi002.stdout:osd.3 smithi002 running (8m) 47s ago 8m 54.8M 2737M 17.2.0 e1d6a67b021e fd99d3764412 2024-09-17T00:22:38.559 INFO:teuthology.orchestra.run.smithi002.stdout:osd.4 smithi099 running (8m) 47s ago 8m 54.5M 2992M 17.2.0 e1d6a67b021e 0af1827b658d 2024-09-17T00:22:38.559 INFO:teuthology.orchestra.run.smithi002.stdout:osd.5 smithi099 running (7m) 47s ago 7m 53.2M 2992M 17.2.0 e1d6a67b021e 4f7ef3aefc58 2024-09-17T00:22:38.560 INFO:teuthology.orchestra.run.smithi002.stdout:osd.6 smithi099 running (7m) 47s ago 7m 49.4M 2992M 17.2.0 e1d6a67b021e 3213f5e6952a 2024-09-17T00:22:38.560 INFO:teuthology.orchestra.run.smithi002.stdout:osd.7 smithi099 running (6m) 47s ago 6m 55.5M 2992M 17.2.0 e1d6a67b021e 63be3a1ed0a1 2024-09-17T00:22:38.560 INFO:teuthology.orchestra.run.smithi002.stdout:prometheus.a smithi099 *:9095 starting - - - - 2024-09-17T00:22:38.560 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.foo.smithi002.uwdhcs smithi002 *:8000 running (4m) 47s ago 4m 73.3M - 17.2.0 e1d6a67b021e 9c35813f459a 2024-09-17T00:22:38.560 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.foo.smithi099.utzfap smithi099 *:8000 running (4m) 47s ago 4m 73.6M - 17.2.0 e1d6a67b021e 34ec989160a6 2024-09-17T00:22:38.560 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.smpl.smithi002.rqnxii smithi002 *:80 running (4m) 47s ago 4m 71.6M - 17.2.0 e1d6a67b021e 0e180d0afb99 2024-09-17T00:22:38.560 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.smpl.smithi099.nzvkpm smithi099 *:80 running (4m) 47s ago 4m 74.4M - 17.2.0 e1d6a67b021e 6a068e82ac72 2024-09-17T00:22:38.717 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:22:38 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[49368]: ts=2024-09-17T00:22:38.659Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd name=CephOSDFlapping index=13 msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.2\", device_class=\"ssd\", hostname=\"smithi002\", instance=\"172.21.15.99:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.2\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"3aad27a2-7489-11ef-bceb-c7b262605968\", cluster_addr=\"172.21.15.2\", device_class=\"ssd\", hostname=\"smithi002\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.2\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-17T00:22:38.717 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:22:38 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: [17/Sep/2024:00:22:38] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-17T00:22:38.717 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:22:38 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: [17/Sep/2024:00:22:38] ENGINE Bus STOPPED 2024-09-17T00:22:38.717 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:22:38 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: [17/Sep/2024:00:22:38] ENGINE Bus STARTING 2024-09-17T00:22:38.717 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:22:38 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: [17/Sep/2024:00:22:38] ENGINE Serving on http://:::9283 2024-09-17T00:22:38.717 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:22:38 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: [17/Sep/2024:00:22:38] ENGINE Bus STARTED 2024-09-17T00:22:38.975 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-09-17T00:22:38.975 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": { 2024-09-17T00:22:38.975 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-09-17T00:22:38.975 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-09-17T00:22:38.975 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": { 2024-09-17T00:22:38.975 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-09-17T00:22:38.975 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 19.1.1-330-gc2ddf972 (c2ddf9721ef254645201cdbabd9c8f49c30a586f) squid (stable)": 1 2024-09-17T00:22:38.975 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-09-17T00:22:38.976 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": { 2024-09-17T00:22:38.976 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-09-17T00:22:38.976 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-09-17T00:22:38.976 INFO:teuthology.orchestra.run.smithi002.stdout: "mds": {}, 2024-09-17T00:22:38.976 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw": { 2024-09-17T00:22:38.976 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-09-17T00:22:38.976 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-09-17T00:22:38.976 INFO:teuthology.orchestra.run.smithi002.stdout: "overall": { 2024-09-17T00:22:38.976 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 16, 2024-09-17T00:22:38.976 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 19.1.1-330-gc2ddf972 (c2ddf9721ef254645201cdbabd9c8f49c30a586f) squid (stable)": 1 2024-09-17T00:22:38.976 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-09-17T00:22:38.976 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-09-17T00:22:39.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:39 smithi002 ceph-mon[28203]: from='client.25183 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:22:39.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:39 smithi002 ceph-mon[28203]: from='client.25189 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:22:39.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:39 smithi002 ceph-mon[28203]: from='client.25195 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:22:39.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:39 smithi002 ceph-mon[28203]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 649 B/s rd, 0 op/s 2024-09-17T00:22:39.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:39 smithi002 ceph-mon[28203]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:22:39.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:39 smithi002 ceph-mon[28203]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-17T00:22:39.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:39 smithi002 ceph-mon[28203]: from='mgr.24986 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-17T00:22:39.230 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:39 smithi002 ceph-mon[28203]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-17T00:22:39.230 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:39 smithi002 ceph-mon[28203]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:22:39.230 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:39 smithi002 ceph-mon[28203]: from='client.? 172.21.15.2:0/2946253562' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:22:39.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:39 smithi002 ceph-mon[23737]: from='client.25183 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:22:39.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:39 smithi002 ceph-mon[23737]: from='client.25189 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:22:39.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:39 smithi002 ceph-mon[23737]: from='client.25195 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:22:39.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:39 smithi002 ceph-mon[23737]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 649 B/s rd, 0 op/s 2024-09-17T00:22:39.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:39 smithi002 ceph-mon[23737]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:22:39.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:39 smithi002 ceph-mon[23737]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-17T00:22:39.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:39 smithi002 ceph-mon[23737]: from='mgr.24986 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-17T00:22:39.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:39 smithi002 ceph-mon[23737]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-17T00:22:39.231 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:39 smithi002 ceph-mon[23737]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:22:39.231 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:39 smithi002 ceph-mon[23737]: from='client.? 172.21.15.2:0/2946253562' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:22:39.354 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-09-17T00:22:39.354 INFO:teuthology.orchestra.run.smithi002.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f", 2024-09-17T00:22:39.354 INFO:teuthology.orchestra.run.smithi002.stdout: "in_progress": true, 2024-09-17T00:22:39.355 INFO:teuthology.orchestra.run.smithi002.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-17T00:22:39.355 INFO:teuthology.orchestra.run.smithi002.stdout: "services_complete": [], 2024-09-17T00:22:39.355 INFO:teuthology.orchestra.run.smithi002.stdout: "progress": "1/23 daemons upgraded", 2024-09-17T00:22:39.355 INFO:teuthology.orchestra.run.smithi002.stdout: "message": "Currently upgrading mgr daemons", 2024-09-17T00:22:39.355 INFO:teuthology.orchestra.run.smithi002.stdout: "is_paused": false 2024-09-17T00:22:39.355 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-09-17T00:22:39.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:39 smithi099 ceph-mon[24594]: from='client.25183 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:22:39.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:39 smithi099 ceph-mon[24594]: from='client.25189 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:22:39.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:39 smithi099 ceph-mon[24594]: from='client.25195 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:22:39.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:39 smithi099 ceph-mon[24594]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 649 B/s rd, 0 op/s 2024-09-17T00:22:39.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:39 smithi099 ceph-mon[24594]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:22:39.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:39 smithi099 ceph-mon[24594]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-17T00:22:39.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:39 smithi099 ceph-mon[24594]: from='mgr.24986 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-17T00:22:39.468 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:39 smithi099 ceph-mon[24594]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-17T00:22:39.468 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:39 smithi099 ceph-mon[24594]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:22:39.468 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:39 smithi099 ceph-mon[24594]: from='client.? 172.21.15.2:0/2946253562' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:22:39.769 INFO:teuthology.orchestra.run.smithi002.stdout:HEALTH_OK 2024-09-17T00:22:40.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:40 smithi099 ceph-mon[24594]: Upgrade: Updating mgr.y 2024-09-17T00:22:40.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:40 smithi099 ceph-mon[24594]: Deploying daemon mgr.y on smithi002 2024-09-17T00:22:40.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:40 smithi099 ceph-mon[24594]: from='client.15321 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:22:40.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:40 smithi099 ceph-mon[24594]: from='client.? 172.21.15.2:0/361616867' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T00:22:40.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:40 smithi002 ceph-mon[28203]: Upgrade: Updating mgr.y 2024-09-17T00:22:40.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:40 smithi002 ceph-mon[28203]: Deploying daemon mgr.y on smithi002 2024-09-17T00:22:40.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:40 smithi002 ceph-mon[28203]: from='client.15321 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:22:40.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:40 smithi002 ceph-mon[28203]: from='client.? 172.21.15.2:0/361616867' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T00:22:40.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:40 smithi002 ceph-mon[23737]: Upgrade: Updating mgr.y 2024-09-17T00:22:40.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:40 smithi002 ceph-mon[23737]: Deploying daemon mgr.y on smithi002 2024-09-17T00:22:40.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:40 smithi002 ceph-mon[23737]: from='client.15321 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:22:40.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:40 smithi002 ceph-mon[23737]: from='client.? 172.21.15.2:0/361616867' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T00:22:40.841 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:22:40 smithi002 systemd[1]: Stopping Ceph mgr.y for 3aad27a2-7489-11ef-bceb-c7b262605968... 2024-09-17T00:22:41.118 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:22:40 smithi002 podman[59059]: 2024-09-17 00:22:40.843443362 +0000 UTC m=+0.147086688 container died d32b04d294d30eed382adf7e696bae32612feb16e79fe2ba590c23be6d622ab7 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_CLEAN=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., ceph=True, CEPH_POINT_RELEASE=-17.2.0, vcs-type=git, release=754, RELEASE=HEAD, io.buildah.version=1.19.8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, architecture=x86_64, vendor=Red Hat, Inc., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base centos centos-stream, distribution-scope=public, version=8, build-date=2022-05-03T08:36:31.336870, com.redhat.component=centos-stream-container, io.openshift.expose-services=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.k8s.display-name=CentOS Stream 8, name=centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git) 2024-09-17T00:22:41.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:41 smithi099 ceph-mon[24594]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 576 B/s rd, 0 op/s 2024-09-17T00:22:41.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:41 smithi002 ceph-mon[23737]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 576 B/s rd, 0 op/s 2024-09-17T00:22:41.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:41 smithi002 ceph-mon[28203]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 576 B/s rd, 0 op/s 2024-09-17T00:22:41.479 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:22:41 smithi002 podman[59059]: 2024-09-17 00:22:41.304454033 +0000 UTC m=+0.608097362 container cleanup d32b04d294d30eed382adf7e696bae32612feb16e79fe2ba590c23be6d622ab7 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y, io.k8s.display-name=CentOS Stream 8, io.openshift.expose-services=, build-date=2022-05-03T08:36:31.336870, GIT_REPO=https://github.com/ceph/ceph-container.git, distribution-scope=public, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, ceph=True, vendor=Red Hat, Inc., com.redhat.component=centos-stream-container, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , name=centos-stream, io.buildah.version=1.19.8, version=8, architecture=x86_64, CEPH_POINT_RELEASE=-17.2.0, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.tags=base centos centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-type=git, release=754, GIT_CLEAN=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, RELEASE=HEAD) 2024-09-17T00:22:41.479 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:22:41 smithi002 bash[59059]: ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y 2024-09-17T00:22:41.979 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:22:41 smithi002 podman[59071]: 2024-09-17 00:22:41.631189523 +0000 UTC m=+0.783627606 container remove d32b04d294d30eed382adf7e696bae32612feb16e79fe2ba590c23be6d622ab7 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.tags=base centos centos-stream, vcs-type=git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_BRANCH=HEAD, name=centos-stream, distribution-scope=public, io.k8s.display-name=CentOS Stream 8, architecture=x86_64, vendor=Red Hat, Inc., GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, ceph=True, RELEASE=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, version=8, io.buildah.version=1.19.8, build-date=2022-05-03T08:36:31.336870, maintainer=Guillaume Abrioux , summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.component=centos-stream-container, release=754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, CEPH_POINT_RELEASE=-17.2.0) 2024-09-17T00:22:41.979 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:22:41 smithi002 systemd[1]: ceph-3aad27a2-7489-11ef-bceb-c7b262605968@mgr.y.service: Main process exited, code=exited, status=143/n/a 2024-09-17T00:22:41.979 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:22:41 smithi002 systemd[1]: ceph-3aad27a2-7489-11ef-bceb-c7b262605968@mgr.y.service: Failed with result 'exit-code'. 2024-09-17T00:22:41.979 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:22:41 smithi002 systemd[1]: Stopped Ceph mgr.y for 3aad27a2-7489-11ef-bceb-c7b262605968. 2024-09-17T00:22:41.979 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:22:41 smithi002 systemd[1]: ceph-3aad27a2-7489-11ef-bceb-c7b262605968@mgr.y.service: Consumed 58.982s CPU time. 2024-09-17T00:22:42.253 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:22:42 smithi002 systemd[1]: Starting Ceph mgr.y for 3aad27a2-7489-11ef-bceb-c7b262605968... 2024-09-17T00:22:42.254 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:22:42 smithi002 podman[59200]: 2024-09-17 00:22:42.113401186 +0000 UTC m=+0.016449550 image pull ebb0615149def4a3179c82c0cc47af75808300abb9f1142788e5e605a7ac0e5b quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f 2024-09-17T00:22:42.510 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:22:42 smithi002 podman[59200]: 2024-09-17 00:22:42.257046352 +0000 UTC m=+0.160094707 container create 8838897e8449c9d8813bf341a6a719e644a65735e3df0b6e29c09bc8ec67ede2 (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, RELEASE=squid-c2ddf97, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-17T00:22:42.510 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:22:42 smithi002 podman[59200]: 2024-09-17 00:22:42.484602867 +0000 UTC m=+0.387651227 container init 8838897e8449c9d8813bf341a6a719e644a65735e3df0b6e29c09bc8ec67ede2 (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=squid-c2ddf97, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2) 2024-09-17T00:22:42.510 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:22:42 smithi002 podman[59200]: 2024-09-17 00:22:42.487798515 +0000 UTC m=+0.390846885 container start 8838897e8449c9d8813bf341a6a719e644a65735e3df0b6e29c09bc8ec67ede2 (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=squid-c2ddf97, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux ) 2024-09-17T00:22:42.844 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:22:42 smithi002 ceph-mgr[59220]: -- 172.21.15.2:0/1238826826 <== mon.1 v2:172.21.15.2:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56434dc9b0e0 con 0x56434dc78800 2024-09-17T00:22:42.844 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:22:42 smithi002 bash[59200]: 8838897e8449c9d8813bf341a6a719e644a65735e3df0b6e29c09bc8ec67ede2 2024-09-17T00:22:42.844 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:22:42 smithi002 systemd[1]: Started Ceph mgr.y for 3aad27a2-7489-11ef-bceb-c7b262605968. 2024-09-17T00:22:42.844 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:22:42 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: 2024-09-17T00:22:42.622+0000 7f8a1c7c9040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-17T00:22:42.845 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:22:42 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: 2024-09-17T00:22:42.685+0000 7f8a1c7c9040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-17T00:22:42.845 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:22:42 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: 2024-09-17T00:22:42.846+0000 7f8a1c7c9040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-17T00:22:43.101 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:22:42 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: 2024-09-17T00:22:42.923+0000 7f8a1c7c9040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-17T00:22:43.386 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:22:43 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: 2024-09-17T00:22:43.103+0000 7f8a1c7c9040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-17T00:22:43.386 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:22:43 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: 2024-09-17T00:22:43.250+0000 7f8a1c7c9040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-17T00:22:43.642 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:22:43 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: 2024-09-17T00:22:43.593+0000 7f8a1c7c9040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-17T00:22:43.967 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:22:43 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[49368]: ts=2024-09-17T00:22:43.505Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes name=CephNodeDiskspaceWarning index=4 msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi002\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi002\", job=\"node\", machine=\"x86_64\", nodename=\"smithi002\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"3aad27a2-7489-11ef-bceb-c7b262605968\", domainname=\"(none)\", instance=\"smithi002\", job=\"node\", machine=\"x86_64\", nodename=\"smithi002\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-17T00:22:43.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:43 smithi099 ceph-mon[24594]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-17T00:22:43.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:43 smithi099 ceph-mon[24594]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:22:43.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:43 smithi099 ceph-mon[24594]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:22:43.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:43 smithi099 ceph-mon[24594]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:22:43.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:43 smithi099 ceph-mon[24594]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:22:43.979 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:22:43 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: 2024-09-17T00:22:43.658+0000 7f8a1c7c9040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-17T00:22:43.979 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:22:43 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: 2024-09-17T00:22:43.865+0000 7f8a1c7c9040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-17T00:22:43.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:43 smithi002 ceph-mon[28203]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-17T00:22:43.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:43 smithi002 ceph-mon[28203]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:22:43.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:43 smithi002 ceph-mon[28203]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:22:43.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:43 smithi002 ceph-mon[28203]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:22:43.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:43 smithi002 ceph-mon[28203]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:22:43.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:43 smithi002 ceph-mon[23737]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-17T00:22:43.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:43 smithi002 ceph-mon[23737]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:22:43.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:43 smithi002 ceph-mon[23737]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:22:43.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:43 smithi002 ceph-mon[23737]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:22:43.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:43 smithi002 ceph-mon[23737]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:22:45.175 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:22:44 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: 2024-09-17T00:22:44.851+0000 7f8a1c7c9040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-17T00:22:45.176 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:22:44 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: 2024-09-17T00:22:44.921+0000 7f8a1c7c9040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-17T00:22:45.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:45 smithi099 ceph-mon[24594]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 862 B/s rd, 0 op/s 2024-09-17T00:22:45.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:45 smithi099 ceph-mon[24594]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T00:22:45.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:45 smithi002 ceph-mon[28203]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 862 B/s rd, 0 op/s 2024-09-17T00:22:45.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:45 smithi002 ceph-mon[28203]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T00:22:45.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:45 smithi002 ceph-mon[23737]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 862 B/s rd, 0 op/s 2024-09-17T00:22:45.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:45 smithi002 ceph-mon[23737]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T00:22:45.980 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:22:45 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: 2024-09-17T00:22:45.706+0000 7f8a1c7c9040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-17T00:22:45.980 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:22:45 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: 2024-09-17T00:22:45.831+0000 7f8a1c7c9040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-17T00:22:46.299 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:22:45 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: 2024-09-17T00:22:45.983+0000 7f8a1c7c9040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-17T00:22:46.299 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:22:46 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: 2024-09-17T00:22:46.047+0000 7f8a1c7c9040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-17T00:22:46.300 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:22:46 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: 2024-09-17T00:22:46.230+0000 7f8a1c7c9040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-17T00:22:46.300 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:22:46 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: 2024-09-17T00:22:46.302+0000 7f8a1c7c9040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-17T00:22:46.556 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:22:46 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: 2024-09-17T00:22:46.365+0000 7f8a1c7c9040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-17T00:22:46.556 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:22:46 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: 2024-09-17T00:22:46.537+0000 7f8a1c7c9040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-17T00:22:46.728 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:46 smithi099 ceph-mon[24594]: from='client.15243 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:22:46.826 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:46 smithi002 ceph-mon[28203]: from='client.15243 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:22:46.826 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:46 smithi002 ceph-mon[23737]: from='client.15243 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:22:46.826 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:22:46 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: 2024-09-17T00:22:46.607+0000 7f8a1c7c9040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-17T00:22:46.826 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:22:46 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: 2024-09-17T00:22:46.679+0000 7f8a1c7c9040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-17T00:22:46.826 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:22:46 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: 2024-09-17T00:22:46.766+0000 7f8a1c7c9040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-17T00:22:46.826 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:22:46 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: 2024-09-17T00:22:46.828+0000 7f8a1c7c9040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-17T00:22:47.093 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:22:46 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: 2024-09-17T00:22:46.893+0000 7f8a1c7c9040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-17T00:22:47.217 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:22:46 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: ::ffff:172.21.15.99 - - [17/Sep/2024:00:22:46] "GET /metrics HTTP/1.1" 200 38134 "" "Prometheus/2.43.0" 2024-09-17T00:22:47.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:47 smithi099 ceph-mon[24594]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:22:47.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:47 smithi099 ceph-mon[24594]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:22:47.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:47 smithi099 ceph-mon[24594]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:22:47.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:47 smithi002 ceph-mon[28203]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:22:47.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:47 smithi002 ceph-mon[28203]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:22:47.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:47 smithi002 ceph-mon[28203]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:22:47.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:47 smithi002 ceph-mon[23737]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:22:47.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:47 smithi002 ceph-mon[23737]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:22:47.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:47 smithi002 ceph-mon[23737]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:22:47.979 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:22:47 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: 2024-09-17T00:22:47.747+0000 7f8a1c7c9040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-17T00:22:47.979 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:22:47 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: [17/Sep/2024:00:22:47] ENGINE Bus STARTING 2024-09-17T00:22:47.980 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:22:47 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: CherryPy Checker: 2024-09-17T00:22:47.980 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:22:47 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: The Application mounted at '' has an empty config. 2024-09-17T00:22:47.980 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:22:47 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: 2024-09-17T00:22:47.980 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:22:47 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: [17/Sep/2024:00:22:47] ENGINE Serving on http://:::9283 2024-09-17T00:22:47.980 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:22:47 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: [17/Sep/2024:00:22:47] ENGINE Bus STARTED 2024-09-17T00:22:48.967 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:22:48 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[49368]: ts=2024-09-17T00:22:48.659Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd name=CephOSDFlapping index=13 msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.2\", device_class=\"ssd\", hostname=\"smithi002\", instance=\"172.21.15.99:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.2\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"3aad27a2-7489-11ef-bceb-c7b262605968\", cluster_addr=\"172.21.15.2\", device_class=\"ssd\", hostname=\"smithi002\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.2\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-17T00:22:48.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:48 smithi099 ceph-mon[24594]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:22:48.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:48 smithi099 ceph-mon[24594]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:22:48.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:48 smithi099 ceph-mon[24594]: Standby manager daemon y restarted 2024-09-17T00:22:48.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:48 smithi099 ceph-mon[24594]: Standby manager daemon y started 2024-09-17T00:22:48.968 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:48 smithi099 ceph-mon[24594]: from='mgr.? 172.21.15.2:0/842744905' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-17T00:22:48.968 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:48 smithi099 ceph-mon[24594]: from='mgr.? 172.21.15.2:0/842744905' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-17T00:22:48.968 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:48 smithi099 ceph-mon[24594]: from='mgr.? 172.21.15.2:0/842744905' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-09-17T00:22:48.968 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:48 smithi099 ceph-mon[24594]: from='mgr.? 172.21.15.2:0/842744905' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-17T00:22:48.968 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:48 smithi099 ceph-mon[24594]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:22:48.968 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:48 smithi099 ceph-mon[24594]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:22:48.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:48 smithi002 ceph-mon[23737]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:22:48.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:48 smithi002 ceph-mon[23737]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:22:48.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:48 smithi002 ceph-mon[23737]: Standby manager daemon y restarted 2024-09-17T00:22:48.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:48 smithi002 ceph-mon[23737]: Standby manager daemon y started 2024-09-17T00:22:48.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:48 smithi002 ceph-mon[23737]: from='mgr.? 172.21.15.2:0/842744905' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-17T00:22:48.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:48 smithi002 ceph-mon[23737]: from='mgr.? 172.21.15.2:0/842744905' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-17T00:22:48.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:48 smithi002 ceph-mon[23737]: from='mgr.? 172.21.15.2:0/842744905' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-09-17T00:22:48.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:48 smithi002 ceph-mon[23737]: from='mgr.? 172.21.15.2:0/842744905' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-17T00:22:48.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:48 smithi002 ceph-mon[23737]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:22:48.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:48 smithi002 ceph-mon[23737]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:22:48.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:48 smithi002 ceph-mon[28203]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:22:48.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:48 smithi002 ceph-mon[28203]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:22:48.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:48 smithi002 ceph-mon[28203]: Standby manager daemon y restarted 2024-09-17T00:22:48.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:48 smithi002 ceph-mon[28203]: Standby manager daemon y started 2024-09-17T00:22:48.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:48 smithi002 ceph-mon[28203]: from='mgr.? 172.21.15.2:0/842744905' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-09-17T00:22:48.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:48 smithi002 ceph-mon[28203]: from='mgr.? 172.21.15.2:0/842744905' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-17T00:22:48.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:48 smithi002 ceph-mon[28203]: from='mgr.? 172.21.15.2:0/842744905' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-09-17T00:22:48.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:48 smithi002 ceph-mon[28203]: from='mgr.? 172.21.15.2:0/842744905' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-17T00:22:48.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:48 smithi002 ceph-mon[28203]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:22:48.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:48 smithi002 ceph-mon[28203]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:22:49.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:49 smithi099 ceph-mon[24594]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:22:49.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:49 smithi099 ceph-mon[24594]: mgrmap e28: x(active, since 65s), standbys: y 2024-09-17T00:22:49.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:49 smithi002 ceph-mon[23737]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:22:49.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:49 smithi002 ceph-mon[23737]: mgrmap e28: x(active, since 65s), standbys: y 2024-09-17T00:22:49.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:49 smithi002 ceph-mon[28203]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:22:49.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:49 smithi002 ceph-mon[28203]: mgrmap e28: x(active, since 65s), standbys: y 2024-09-17T00:22:51.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:51 smithi099 ceph-mon[24594]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:22:51.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:51 smithi099 ceph-mon[24594]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:22:51.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:51 smithi099 ceph-mon[24594]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:22:51.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:51 smithi099 ceph-mon[24594]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:22:51.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:51 smithi099 ceph-mon[24594]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:22:51.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:51 smithi099 ceph-mon[24594]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:22:51.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:51 smithi002 ceph-mon[23737]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:22:51.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:51 smithi002 ceph-mon[23737]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:22:51.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:51 smithi002 ceph-mon[23737]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:22:51.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:51 smithi002 ceph-mon[23737]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:22:51.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:51 smithi002 ceph-mon[23737]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:22:51.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:51 smithi002 ceph-mon[23737]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:22:51.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:51 smithi002 ceph-mon[28203]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:22:51.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:51 smithi002 ceph-mon[28203]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:22:51.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:51 smithi002 ceph-mon[28203]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:22:51.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:51 smithi002 ceph-mon[28203]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:22:51.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:51 smithi002 ceph-mon[28203]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:22:51.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:51 smithi002 ceph-mon[28203]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:22:52.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:52 smithi099 ceph-mon[24594]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi002.vmouem", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-17T00:22:52.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:52 smithi099 ceph-mon[24594]: from='mgr.24986 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi002.vmouem", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-17T00:22:52.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:52 smithi099 ceph-mon[24594]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:22:52.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:52 smithi002 ceph-mon[23737]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi002.vmouem", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-17T00:22:52.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:52 smithi002 ceph-mon[23737]: from='mgr.24986 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi002.vmouem", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-17T00:22:52.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:52 smithi002 ceph-mon[23737]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:22:52.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:52 smithi002 ceph-mon[28203]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi002.vmouem", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-17T00:22:52.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:52 smithi002 ceph-mon[28203]: from='mgr.24986 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi002.vmouem", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-17T00:22:52.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:52 smithi002 ceph-mon[28203]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:22:53.884 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:53 smithi002 ceph-mon[23737]: Reconfiguring iscsi.foo.smithi002.vmouem (dependencies changed)... 2024-09-17T00:22:53.884 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:53 smithi002 ceph-mon[23737]: Reconfiguring daemon iscsi.foo.smithi002.vmouem on smithi002 2024-09-17T00:22:53.884 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:53 smithi002 ceph-mon[23737]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:22:53.884 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:53 smithi002 ceph-mon[28203]: Reconfiguring iscsi.foo.smithi002.vmouem (dependencies changed)... 2024-09-17T00:22:53.884 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:53 smithi002 ceph-mon[28203]: Reconfiguring daemon iscsi.foo.smithi002.vmouem on smithi002 2024-09-17T00:22:53.884 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:53 smithi002 ceph-mon[28203]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:22:53.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:53 smithi099 ceph-mon[24594]: Reconfiguring iscsi.foo.smithi002.vmouem (dependencies changed)... 2024-09-17T00:22:53.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:53 smithi099 ceph-mon[24594]: Reconfiguring daemon iscsi.foo.smithi002.vmouem on smithi002 2024-09-17T00:22:53.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:53 smithi099 ceph-mon[24594]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:22:53.967 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:22:53 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[49368]: ts=2024-09-17T00:22:53.505Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes name=CephNodeDiskspaceWarning index=4 msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi002\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi002\", job=\"node\", machine=\"x86_64\", nodename=\"smithi002\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"3aad27a2-7489-11ef-bceb-c7b262605968\", domainname=\"(none)\", instance=\"smithi002\", job=\"node\", machine=\"x86_64\", nodename=\"smithi002\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-17T00:22:55.807 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:55 smithi002 ceph-mon[23737]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:22:55.807 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:55 smithi002 ceph-mon[28203]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:22:55.966 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:55 smithi099 ceph-mon[24594]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:22:56.987 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:22:56 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: ::ffff:172.21.15.99 - - [17/Sep/2024:00:22:56] "GET /metrics HTTP/1.1" 200 38134 "" "Prometheus/2.43.0" 2024-09-17T00:22:56.987 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:22:56 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: 2024-09-17T00:22:56.855+0000 7f0e20e70640 -1 mgr handle_mgr_map I was active but no longer am 2024-09-17T00:22:56.987 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:22:56 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: ignoring --setuser ceph since I am not root 2024-09-17T00:22:56.987 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:22:56 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: ignoring --setgroup ceph since I am not root 2024-09-17T00:22:56.987 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:22:56 smithi099 ceph-mgr[45660]: -- 172.21.15.99:0/840027904 <== mon.2 v2:172.21.15.99:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55cf8db6d0e0 con 0x55cf8db4b400 2024-09-17T00:22:56.988 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:56 smithi099 ceph-mon[24594]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:22:56.988 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:56 smithi099 ceph-mon[24594]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:22:56.988 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:56 smithi099 ceph-mon[24594]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-09-17T00:22:56.988 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:56 smithi099 ceph-mon[24594]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-09-17T00:22:56.988 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:56 smithi099 ceph-mon[24594]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-09-17T00:22:56.988 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:56 smithi099 ceph-mon[24594]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-09-17T00:22:56.988 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:56 smithi099 ceph-mon[24594]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T00:22:56.988 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:56 smithi099 ceph-mon[24594]: Upgrade: Need to upgrade myself (mgr.x) 2024-09-17T00:22:56.988 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:56 smithi099 ceph-mon[24594]: Upgrade: Need to upgrade myself (mgr.x) 2024-09-17T00:22:56.988 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:56 smithi099 ceph-mon[24594]: Failing over to other MGR 2024-09-17T00:22:56.988 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:56 smithi099 ceph-mon[24594]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-09-17T00:22:56.988 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:56 smithi099 ceph-mon[24594]: from='mgr.24986 ' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-09-17T00:22:56.989 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:56 smithi099 ceph-mon[24594]: osdmap e105: 8 total, 8 up, 8 in 2024-09-17T00:22:56.989 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:56 smithi099 ceph-mon[24594]: from='client.? 172.21.15.2:0/3852221033' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-17T00:22:56.989 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:56 smithi099 ceph-mon[24594]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 818 B/s rd, 0 op/s 2024-09-17T00:22:57.124 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:22:56 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: [17/Sep/2024:00:22:56] ENGINE Bus STOPPING 2024-09-17T00:22:57.125 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:22:56 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: [17/Sep/2024:00:22:56] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-17T00:22:57.125 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:22:56 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: [17/Sep/2024:00:22:56] ENGINE Bus STOPPED 2024-09-17T00:22:57.125 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:22:57 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: [17/Sep/2024:00:22:57] ENGINE Bus STARTING 2024-09-17T00:22:57.125 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:22:57 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: [17/Sep/2024:00:22:57] ENGINE Serving on http://:::9283 2024-09-17T00:22:57.125 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:56 smithi002 ceph-mon[23737]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:22:57.125 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:56 smithi002 ceph-mon[23737]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:22:57.125 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:56 smithi002 ceph-mon[23737]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-09-17T00:22:57.125 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:56 smithi002 ceph-mon[23737]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-09-17T00:22:57.125 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:56 smithi002 ceph-mon[23737]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-09-17T00:22:57.125 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:56 smithi002 ceph-mon[23737]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-09-17T00:22:57.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:56 smithi002 ceph-mon[23737]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T00:22:57.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:56 smithi002 ceph-mon[23737]: Upgrade: Need to upgrade myself (mgr.x) 2024-09-17T00:22:57.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:56 smithi002 ceph-mon[23737]: Upgrade: Need to upgrade myself (mgr.x) 2024-09-17T00:22:57.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:56 smithi002 ceph-mon[23737]: Failing over to other MGR 2024-09-17T00:22:57.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:56 smithi002 ceph-mon[23737]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-09-17T00:22:57.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:56 smithi002 ceph-mon[23737]: from='mgr.24986 ' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-09-17T00:22:57.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:56 smithi002 ceph-mon[23737]: osdmap e105: 8 total, 8 up, 8 in 2024-09-17T00:22:57.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:56 smithi002 ceph-mon[23737]: from='client.? 172.21.15.2:0/3852221033' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-17T00:22:57.126 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:56 smithi002 ceph-mon[23737]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 818 B/s rd, 0 op/s 2024-09-17T00:22:57.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:56 smithi002 ceph-mon[28203]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:22:57.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:56 smithi002 ceph-mon[28203]: from='mgr.24986 ' entity='mgr.x' 2024-09-17T00:22:57.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:56 smithi002 ceph-mon[28203]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-09-17T00:22:57.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:56 smithi002 ceph-mon[28203]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-09-17T00:22:57.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:56 smithi002 ceph-mon[28203]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-09-17T00:22:57.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:56 smithi002 ceph-mon[28203]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-09-17T00:22:57.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:56 smithi002 ceph-mon[28203]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T00:22:57.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:56 smithi002 ceph-mon[28203]: Upgrade: Need to upgrade myself (mgr.x) 2024-09-17T00:22:57.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:56 smithi002 ceph-mon[28203]: Upgrade: Need to upgrade myself (mgr.x) 2024-09-17T00:22:57.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:56 smithi002 ceph-mon[28203]: Failing over to other MGR 2024-09-17T00:22:57.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:56 smithi002 ceph-mon[28203]: from='mgr.24986 172.21.15.99:0/3268374840' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-09-17T00:22:57.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:56 smithi002 ceph-mon[28203]: from='mgr.24986 ' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-09-17T00:22:57.127 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:56 smithi002 ceph-mon[28203]: osdmap e105: 8 total, 8 up, 8 in 2024-09-17T00:22:57.128 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:56 smithi002 ceph-mon[28203]: from='client.? 172.21.15.2:0/3852221033' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-17T00:22:57.128 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:56 smithi002 ceph-mon[28203]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 818 B/s rd, 0 op/s 2024-09-17T00:22:57.282 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:22:57 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: 2024-09-17T00:22:57.016+0000 7f5646056040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-17T00:22:57.282 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:22:57 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: 2024-09-17T00:22:57.074+0000 7f5646056040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-17T00:22:57.282 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:22:57 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: 2024-09-17T00:22:57.220+0000 7f5646056040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-17T00:22:57.282 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:22:57 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: 2024-09-17T00:22:57.284+0000 7f5646056040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-17T00:22:57.382 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:22:57 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: [17/Sep/2024:00:22:57] ENGINE Bus STARTED 2024-09-17T00:22:57.581 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:22:57 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: 2024-09-17T00:22:57.435+0000 7f5646056040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-17T00:22:57.581 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:22:57 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: 2024-09-17T00:22:57.584+0000 7f5646056040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-17T00:22:58.186 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:22:57 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: 2024-09-17T00:22:57.931+0000 7f5646056040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-17T00:22:58.186 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:22:57 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: 2024-09-17T00:22:57.993+0000 7f5646056040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-17T00:22:58.186 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:57 smithi099 ceph-mon[24594]: from='mgr.24986 ' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2024-09-17T00:22:58.186 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:57 smithi099 ceph-mon[24594]: mgrmap e29: y(active, starting, since 0.952338s) 2024-09-17T00:22:58.186 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:57 smithi099 ceph-mon[24594]: from='mgr.15333 172.21.15.2:0/842744905' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T00:22:58.186 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:57 smithi099 ceph-mon[24594]: from='mgr.15333 172.21.15.2:0/842744905' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T00:22:58.186 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:57 smithi099 ceph-mon[24594]: from='mgr.15333 172.21.15.2:0/842744905' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T00:22:58.186 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:57 smithi099 ceph-mon[24594]: from='mgr.15333 172.21.15.2:0/842744905' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-17T00:22:58.187 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:57 smithi099 ceph-mon[24594]: from='mgr.15333 172.21.15.2:0/842744905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T00:22:58.187 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:57 smithi099 ceph-mon[24594]: from='mgr.15333 172.21.15.2:0/842744905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T00:22:58.187 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:57 smithi099 ceph-mon[24594]: from='mgr.15333 172.21.15.2:0/842744905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T00:22:58.187 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:57 smithi099 ceph-mon[24594]: from='mgr.15333 172.21.15.2:0/842744905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T00:22:58.187 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:57 smithi099 ceph-mon[24594]: from='mgr.15333 172.21.15.2:0/842744905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T00:22:58.187 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:57 smithi099 ceph-mon[24594]: from='mgr.15333 172.21.15.2:0/842744905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T00:22:58.187 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:57 smithi099 ceph-mon[24594]: from='mgr.15333 172.21.15.2:0/842744905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T00:22:58.187 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:57 smithi099 ceph-mon[24594]: from='mgr.15333 172.21.15.2:0/842744905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T00:22:58.187 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:57 smithi099 ceph-mon[24594]: from='mgr.15333 172.21.15.2:0/842744905' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-17T00:22:58.187 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:57 smithi099 ceph-mon[24594]: from='mgr.15333 172.21.15.2:0/842744905' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-17T00:22:58.187 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:57 smithi099 ceph-mon[24594]: from='mgr.15333 172.21.15.2:0/842744905' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-17T00:22:58.187 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:57 smithi099 ceph-mon[24594]: Manager daemon y is now available 2024-09-17T00:22:58.187 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:57 smithi099 ceph-mon[24594]: from='mgr.15333 172.21.15.2:0/842744905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T00:22:58.187 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:57 smithi099 ceph-mon[24594]: from='mgr.15333 172.21.15.2:0/842744905' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T00:22:58.188 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:57 smithi099 ceph-mon[24594]: from='mgr.15333 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T00:22:58.188 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:57 smithi099 ceph-mon[24594]: from='mgr.15333 172.21.15.2:0/842744905' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T00:22:58.188 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:57 smithi099 ceph-mon[24594]: from='mgr.15333 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T00:22:58.230 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:57 smithi002 ceph-mon[28203]: from='mgr.24986 ' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2024-09-17T00:22:58.230 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:57 smithi002 ceph-mon[28203]: mgrmap e29: y(active, starting, since 0.952338s) 2024-09-17T00:22:58.230 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:57 smithi002 ceph-mon[28203]: from='mgr.15333 172.21.15.2:0/842744905' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T00:22:58.230 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:57 smithi002 ceph-mon[28203]: from='mgr.15333 172.21.15.2:0/842744905' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T00:22:58.230 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:57 smithi002 ceph-mon[28203]: from='mgr.15333 172.21.15.2:0/842744905' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T00:22:58.230 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:57 smithi002 ceph-mon[28203]: from='mgr.15333 172.21.15.2:0/842744905' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-17T00:22:58.230 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:57 smithi002 ceph-mon[28203]: from='mgr.15333 172.21.15.2:0/842744905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T00:22:58.230 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:57 smithi002 ceph-mon[28203]: from='mgr.15333 172.21.15.2:0/842744905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T00:22:58.230 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:57 smithi002 ceph-mon[28203]: from='mgr.15333 172.21.15.2:0/842744905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T00:22:58.230 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:57 smithi002 ceph-mon[28203]: from='mgr.15333 172.21.15.2:0/842744905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T00:22:58.230 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:57 smithi002 ceph-mon[28203]: from='mgr.15333 172.21.15.2:0/842744905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T00:22:58.230 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:57 smithi002 ceph-mon[28203]: from='mgr.15333 172.21.15.2:0/842744905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T00:22:58.231 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:57 smithi002 ceph-mon[28203]: from='mgr.15333 172.21.15.2:0/842744905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T00:22:58.231 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:57 smithi002 ceph-mon[28203]: from='mgr.15333 172.21.15.2:0/842744905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T00:22:58.231 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:57 smithi002 ceph-mon[28203]: from='mgr.15333 172.21.15.2:0/842744905' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-17T00:22:58.231 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:57 smithi002 ceph-mon[28203]: from='mgr.15333 172.21.15.2:0/842744905' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-17T00:22:58.231 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:57 smithi002 ceph-mon[28203]: from='mgr.15333 172.21.15.2:0/842744905' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-17T00:22:58.231 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:57 smithi002 ceph-mon[28203]: Manager daemon y is now available 2024-09-17T00:22:58.231 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:57 smithi002 ceph-mon[28203]: from='mgr.15333 172.21.15.2:0/842744905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T00:22:58.231 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:57 smithi002 ceph-mon[28203]: from='mgr.15333 172.21.15.2:0/842744905' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T00:22:58.231 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:57 smithi002 ceph-mon[28203]: from='mgr.15333 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T00:22:58.231 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:57 smithi002 ceph-mon[28203]: from='mgr.15333 172.21.15.2:0/842744905' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T00:22:58.231 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:57 smithi002 ceph-mon[28203]: from='mgr.15333 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T00:22:58.232 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:57 smithi002 ceph-mon[23737]: from='mgr.24986 ' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2024-09-17T00:22:58.232 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:57 smithi002 ceph-mon[23737]: mgrmap e29: y(active, starting, since 0.952338s) 2024-09-17T00:22:58.232 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:57 smithi002 ceph-mon[23737]: from='mgr.15333 172.21.15.2:0/842744905' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T00:22:58.232 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:57 smithi002 ceph-mon[23737]: from='mgr.15333 172.21.15.2:0/842744905' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T00:22:58.232 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:57 smithi002 ceph-mon[23737]: from='mgr.15333 172.21.15.2:0/842744905' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T00:22:58.232 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:57 smithi002 ceph-mon[23737]: from='mgr.15333 172.21.15.2:0/842744905' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-17T00:22:58.232 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:57 smithi002 ceph-mon[23737]: from='mgr.15333 172.21.15.2:0/842744905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T00:22:58.232 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:57 smithi002 ceph-mon[23737]: from='mgr.15333 172.21.15.2:0/842744905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T00:22:58.232 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:57 smithi002 ceph-mon[23737]: from='mgr.15333 172.21.15.2:0/842744905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T00:22:58.232 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:57 smithi002 ceph-mon[23737]: from='mgr.15333 172.21.15.2:0/842744905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T00:22:58.232 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:57 smithi002 ceph-mon[23737]: from='mgr.15333 172.21.15.2:0/842744905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T00:22:58.232 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:57 smithi002 ceph-mon[23737]: from='mgr.15333 172.21.15.2:0/842744905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T00:22:58.232 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:57 smithi002 ceph-mon[23737]: from='mgr.15333 172.21.15.2:0/842744905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T00:22:58.232 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:57 smithi002 ceph-mon[23737]: from='mgr.15333 172.21.15.2:0/842744905' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T00:22:58.233 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:57 smithi002 ceph-mon[23737]: from='mgr.15333 172.21.15.2:0/842744905' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-17T00:22:58.233 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:57 smithi002 ceph-mon[23737]: from='mgr.15333 172.21.15.2:0/842744905' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-17T00:22:58.233 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:57 smithi002 ceph-mon[23737]: from='mgr.15333 172.21.15.2:0/842744905' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-17T00:22:58.233 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:57 smithi002 ceph-mon[23737]: Manager daemon y is now available 2024-09-17T00:22:58.233 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:57 smithi002 ceph-mon[23737]: from='mgr.15333 172.21.15.2:0/842744905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T00:22:58.233 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:57 smithi002 ceph-mon[23737]: from='mgr.15333 172.21.15.2:0/842744905' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T00:22:58.233 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:57 smithi002 ceph-mon[23737]: from='mgr.15333 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T00:22:58.233 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:57 smithi002 ceph-mon[23737]: from='mgr.15333 172.21.15.2:0/842744905' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T00:22:58.233 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:57 smithi002 ceph-mon[23737]: from='mgr.15333 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T00:22:58.466 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:22:58 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: 2024-09-17T00:22:58.188+0000 7f5646056040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-17T00:22:58.909 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:22:58 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[49368]: ts=2024-09-17T00:22:58.660Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd name=CephOSDFlapping index=13 msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.2\", device_class=\"ssd\", hostname=\"smithi002\", instance=\"172.21.15.99:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.2\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"3aad27a2-7489-11ef-bceb-c7b262605968\", cluster_addr=\"172.21.15.2\", device_class=\"ssd\", hostname=\"smithi002\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.2\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-17T00:22:58.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:58 smithi002 ceph-mon[28203]: mgrmap e30: y(active, since 2s) 2024-09-17T00:22:58.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:58 smithi002 ceph-mon[28203]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:22:58.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:58 smithi002 ceph-mon[23737]: mgrmap e30: y(active, since 2s) 2024-09-17T00:22:58.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:58 smithi002 ceph-mon[23737]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:22:58.980 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:22:58 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: 2024-09-17T00:22:58.722+0000 7f8978fc3640 -1 log_channel(cephadm) log [ERR] : [17/Sep/2024:00:22:58] ENGINE Error in HTTPServer.serve 2024-09-17T00:22:58.980 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:22:58 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: Traceback (most recent call last): 2024-09-17T00:22:58.980 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:22:58 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-17T00:22:58.980 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:22:58 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: self._connections.run(self.expiration_interval) 2024-09-17T00:22:58.980 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:22:58 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-17T00:22:58.980 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:22:58 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: self._run(expiration_interval) 2024-09-17T00:22:58.980 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:22:58 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-17T00:22:58.980 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:22:58 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: new_conn = self._from_server_socket(self.server.socket) 2024-09-17T00:22:58.980 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:22:58 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-17T00:22:58.980 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:22:58 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-17T00:22:58.981 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:22:58 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-17T00:22:58.981 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:22:58 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: s = self.context.wrap_socket( 2024-09-17T00:22:58.981 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:22:58 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-17T00:22:58.981 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:22:58 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: return self.sslsocket_class._create( 2024-09-17T00:22:58.981 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:22:58 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-17T00:22:58.981 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:22:58 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: self.do_handshake() 2024-09-17T00:22:58.981 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:22:58 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-17T00:22:58.981 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:22:58 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: self._sslobj.do_handshake() 2024-09-17T00:22:58.981 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:22:58 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-17T00:22:58.981 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:22:58 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: 2024-09-17T00:22:59.172 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:22:59 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: 2024-09-17T00:22:59.106+0000 7f5646056040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-17T00:22:59.173 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:58 smithi099 ceph-mon[24594]: mgrmap e30: y(active, since 2s) 2024-09-17T00:22:59.173 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:58 smithi099 ceph-mon[24594]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:22:59.466 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:22:59 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: 2024-09-17T00:22:59.174+0000 7f5646056040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-17T00:22:59.933 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:59 smithi002 ceph-mon[28203]: [17/Sep/2024:00:22:58] ENGINE Bus STARTING 2024-09-17T00:22:59.933 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:59 smithi002 ceph-mon[28203]: [17/Sep/2024:00:22:58] ENGINE Serving on https://172.21.15.2:7150 2024-09-17T00:22:59.934 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:59 smithi002 ceph-mon[28203]: [17/Sep/2024:00:22:58] ENGINE Error in HTTPServer.serve 2024-09-17T00:22:59.934 INFO:journalctl@ceph.mon.c.smithi002.stdout: Traceback (most recent call last): 2024-09-17T00:22:59.934 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-17T00:22:59.934 INFO:journalctl@ceph.mon.c.smithi002.stdout: self._connections.run(self.expiration_interval) 2024-09-17T00:22:59.934 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-17T00:22:59.934 INFO:journalctl@ceph.mon.c.smithi002.stdout: self._run(expiration_interval) 2024-09-17T00:22:59.934 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-17T00:22:59.934 INFO:journalctl@ceph.mon.c.smithi002.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-17T00:22:59.934 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-17T00:22:59.934 INFO:journalctl@ceph.mon.c.smithi002.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-17T00:22:59.934 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-17T00:22:59.935 INFO:journalctl@ceph.mon.c.smithi002.stdout: s = self.context.wrap_socket( 2024-09-17T00:22:59.935 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-17T00:22:59.935 INFO:journalctl@ceph.mon.c.smithi002.stdout: return self.sslsocket_class._create( 2024-09-17T00:22:59.935 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-17T00:22:59.935 INFO:journalctl@ceph.mon.c.smithi002.stdout: self.do_handshake() 2024-09-17T00:22:59.935 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-17T00:22:59.935 INFO:journalctl@ceph.mon.c.smithi002.stdout: self._sslobj.do_handshake() 2024-09-17T00:22:59.935 INFO:journalctl@ceph.mon.c.smithi002.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-17T00:22:59.935 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:59 smithi002 ceph-mon[28203]: [17/Sep/2024:00:22:58] ENGINE Serving on http://172.21.15.2:8765 2024-09-17T00:22:59.935 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:59 smithi002 ceph-mon[28203]: [17/Sep/2024:00:22:58] ENGINE Bus STARTED 2024-09-17T00:22:59.935 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:22:59 smithi002 ceph-mon[28203]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:22:59.935 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:59 smithi002 ceph-mon[23737]: [17/Sep/2024:00:22:58] ENGINE Bus STARTING 2024-09-17T00:22:59.936 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:59 smithi002 ceph-mon[23737]: [17/Sep/2024:00:22:58] ENGINE Serving on https://172.21.15.2:7150 2024-09-17T00:22:59.936 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:59 smithi002 ceph-mon[23737]: [17/Sep/2024:00:22:58] ENGINE Error in HTTPServer.serve 2024-09-17T00:22:59.936 INFO:journalctl@ceph.mon.a.smithi002.stdout: Traceback (most recent call last): 2024-09-17T00:22:59.936 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-17T00:22:59.936 INFO:journalctl@ceph.mon.a.smithi002.stdout: self._connections.run(self.expiration_interval) 2024-09-17T00:22:59.936 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-17T00:22:59.936 INFO:journalctl@ceph.mon.a.smithi002.stdout: self._run(expiration_interval) 2024-09-17T00:22:59.936 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-17T00:22:59.936 INFO:journalctl@ceph.mon.a.smithi002.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-17T00:22:59.937 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-17T00:22:59.937 INFO:journalctl@ceph.mon.a.smithi002.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-17T00:22:59.937 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-17T00:22:59.937 INFO:journalctl@ceph.mon.a.smithi002.stdout: s = self.context.wrap_socket( 2024-09-17T00:22:59.937 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-17T00:22:59.937 INFO:journalctl@ceph.mon.a.smithi002.stdout: return self.sslsocket_class._create( 2024-09-17T00:22:59.937 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-17T00:22:59.937 INFO:journalctl@ceph.mon.a.smithi002.stdout: self.do_handshake() 2024-09-17T00:22:59.937 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-17T00:22:59.937 INFO:journalctl@ceph.mon.a.smithi002.stdout: self._sslobj.do_handshake() 2024-09-17T00:22:59.937 INFO:journalctl@ceph.mon.a.smithi002.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-17T00:22:59.937 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:59 smithi002 ceph-mon[23737]: [17/Sep/2024:00:22:58] ENGINE Serving on http://172.21.15.2:8765 2024-09-17T00:22:59.938 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:59 smithi002 ceph-mon[23737]: [17/Sep/2024:00:22:58] ENGINE Bus STARTED 2024-09-17T00:22:59.938 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:22:59 smithi002 ceph-mon[23737]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:22:59.938 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:22:59 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: 2024-09-17T00:22:59.930+0000 7f5646056040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-17T00:22:59.938 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:59 smithi099 ceph-mon[24594]: [17/Sep/2024:00:22:58] ENGINE Bus STARTING 2024-09-17T00:22:59.938 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:59 smithi099 ceph-mon[24594]: [17/Sep/2024:00:22:58] ENGINE Serving on https://172.21.15.2:7150 2024-09-17T00:22:59.938 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:59 smithi099 ceph-mon[24594]: [17/Sep/2024:00:22:58] ENGINE Error in HTTPServer.serve 2024-09-17T00:22:59.938 INFO:journalctl@ceph.mon.b.smithi099.stdout: Traceback (most recent call last): 2024-09-17T00:22:59.939 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-17T00:22:59.939 INFO:journalctl@ceph.mon.b.smithi099.stdout: self._connections.run(self.expiration_interval) 2024-09-17T00:22:59.939 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-17T00:22:59.939 INFO:journalctl@ceph.mon.b.smithi099.stdout: self._run(expiration_interval) 2024-09-17T00:22:59.939 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-17T00:22:59.939 INFO:journalctl@ceph.mon.b.smithi099.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-17T00:22:59.939 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-17T00:22:59.939 INFO:journalctl@ceph.mon.b.smithi099.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-17T00:22:59.939 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-17T00:22:59.939 INFO:journalctl@ceph.mon.b.smithi099.stdout: s = self.context.wrap_socket( 2024-09-17T00:22:59.939 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-17T00:22:59.939 INFO:journalctl@ceph.mon.b.smithi099.stdout: return self.sslsocket_class._create( 2024-09-17T00:22:59.939 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-17T00:22:59.939 INFO:journalctl@ceph.mon.b.smithi099.stdout: self.do_handshake() 2024-09-17T00:22:59.940 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-17T00:22:59.940 INFO:journalctl@ceph.mon.b.smithi099.stdout: self._sslobj.do_handshake() 2024-09-17T00:22:59.940 INFO:journalctl@ceph.mon.b.smithi099.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-17T00:22:59.940 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:59 smithi099 ceph-mon[24594]: [17/Sep/2024:00:22:58] ENGINE Serving on http://172.21.15.2:8765 2024-09-17T00:22:59.940 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:59 smithi099 ceph-mon[24594]: [17/Sep/2024:00:22:58] ENGINE Bus STARTED 2024-09-17T00:22:59.940 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:22:59 smithi099 ceph-mon[24594]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:23:00.200 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:23:00 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: 2024-09-17T00:23:00.051+0000 7f5646056040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-17T00:23:00.465 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:23:00 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: 2024-09-17T00:23:00.203+0000 7f5646056040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-17T00:23:00.465 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:23:00 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: 2024-09-17T00:23:00.270+0000 7f5646056040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-17T00:23:00.465 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:23:00 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: 2024-09-17T00:23:00.418+0000 7f5646056040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-17T00:23:00.717 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:23:00 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: 2024-09-17T00:23:00.492+0000 7f5646056040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-17T00:23:00.717 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:23:00 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: 2024-09-17T00:23:00.553+0000 7f5646056040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-17T00:23:00.717 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:23:00 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: 2024-09-17T00:23:00.714+0000 7f5646056040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-17T00:23:01.066 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:00 smithi099 ceph-mon[24594]: mgrmap e31: y(active, since 4s) 2024-09-17T00:23:01.066 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:23:00 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: 2024-09-17T00:23:00.781+0000 7f5646056040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-17T00:23:01.066 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:23:00 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: 2024-09-17T00:23:00.848+0000 7f5646056040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-17T00:23:01.067 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:23:00 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: 2024-09-17T00:23:00.936+0000 7f5646056040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-17T00:23:01.067 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:23:01 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: 2024-09-17T00:23:01.009+0000 7f5646056040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-17T00:23:01.067 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:23:01 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: 2024-09-17T00:23:01.069+0000 7f5646056040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-17T00:23:01.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:00 smithi002 ceph-mon[28203]: mgrmap e31: y(active, since 4s) 2024-09-17T00:23:01.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:00 smithi002 ceph-mon[23737]: mgrmap e31: y(active, since 4s) 2024-09-17T00:23:02.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:01 smithi099 ceph-mon[24594]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:23:02.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:01 smithi099 ceph-mon[24594]: from='mgr.15333 ' entity='mgr.y' 2024-09-17T00:23:02.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:01 smithi099 ceph-mon[24594]: from='mgr.15333 ' entity='mgr.y' 2024-09-17T00:23:02.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:01 smithi099 ceph-mon[24594]: Standby manager daemon x started 2024-09-17T00:23:02.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:01 smithi099 ceph-mon[24594]: from='mgr.? 172.21.15.99:0/546084788' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-17T00:23:02.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:01 smithi099 ceph-mon[24594]: from='mgr.? 172.21.15.99:0/546084788' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-17T00:23:02.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:01 smithi099 ceph-mon[24594]: from='mgr.? 172.21.15.99:0/546084788' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-17T00:23:02.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:01 smithi099 ceph-mon[24594]: from='mgr.? 172.21.15.99:0/546084788' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-17T00:23:02.218 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:23:01 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: 2024-09-17T00:23:01.893+0000 7f5646056040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-17T00:23:02.218 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:23:01 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: [17/Sep/2024:00:23:01] ENGINE Bus STARTING 2024-09-17T00:23:02.218 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:23:01 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: CherryPy Checker: 2024-09-17T00:23:02.218 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:23:01 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: The Application mounted at '' has an empty config. 2024-09-17T00:23:02.218 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:23:01 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: 2024-09-17T00:23:02.218 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:23:02 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: [17/Sep/2024:00:23:02] ENGINE Serving on http://:::9283 2024-09-17T00:23:02.218 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:23:02 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: [17/Sep/2024:00:23:02] ENGINE Bus STARTED 2024-09-17T00:23:02.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:01 smithi002 ceph-mon[28203]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:23:02.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:01 smithi002 ceph-mon[28203]: from='mgr.15333 ' entity='mgr.y' 2024-09-17T00:23:02.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:01 smithi002 ceph-mon[28203]: from='mgr.15333 ' entity='mgr.y' 2024-09-17T00:23:02.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:01 smithi002 ceph-mon[28203]: Standby manager daemon x started 2024-09-17T00:23:02.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:01 smithi002 ceph-mon[28203]: from='mgr.? 172.21.15.99:0/546084788' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-17T00:23:02.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:01 smithi002 ceph-mon[28203]: from='mgr.? 172.21.15.99:0/546084788' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-17T00:23:02.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:01 smithi002 ceph-mon[28203]: from='mgr.? 172.21.15.99:0/546084788' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-17T00:23:02.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:01 smithi002 ceph-mon[28203]: from='mgr.? 172.21.15.99:0/546084788' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-17T00:23:02.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:01 smithi002 ceph-mon[23737]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:23:02.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:01 smithi002 ceph-mon[23737]: from='mgr.15333 ' entity='mgr.y' 2024-09-17T00:23:02.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:01 smithi002 ceph-mon[23737]: from='mgr.15333 ' entity='mgr.y' 2024-09-17T00:23:02.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:01 smithi002 ceph-mon[23737]: Standby manager daemon x started 2024-09-17T00:23:02.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:01 smithi002 ceph-mon[23737]: from='mgr.? 172.21.15.99:0/546084788' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-17T00:23:02.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:01 smithi002 ceph-mon[23737]: from='mgr.? 172.21.15.99:0/546084788' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-17T00:23:02.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:01 smithi002 ceph-mon[23737]: from='mgr.? 172.21.15.99:0/546084788' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-17T00:23:02.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:01 smithi002 ceph-mon[23737]: from='mgr.? 172.21.15.99:0/546084788' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-17T00:23:03.718 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:23:03 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[49368]: ts=2024-09-17T00:23:03.505Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes name=CephNodeDiskspaceWarning index=4 msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi002\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi002\", job=\"node\", machine=\"x86_64\", nodename=\"smithi002\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"3aad27a2-7489-11ef-bceb-c7b262605968\", domainname=\"(none)\", instance=\"smithi002\", job=\"node\", machine=\"x86_64\", nodename=\"smithi002\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-17T00:23:03.718 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:03 smithi099 ceph-mon[24594]: from='mgr.15333 ' entity='mgr.y' 2024-09-17T00:23:03.718 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:03 smithi099 ceph-mon[24594]: from='mgr.15333 ' entity='mgr.y' 2024-09-17T00:23:03.719 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:03 smithi099 ceph-mon[24594]: from='mgr.15333 ' entity='mgr.y' 2024-09-17T00:23:03.719 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:03 smithi099 ceph-mon[24594]: from='mgr.15333 ' entity='mgr.y' 2024-09-17T00:23:03.719 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:03 smithi099 ceph-mon[24594]: mgrmap e32: y(active, since 6s), standbys: x 2024-09-17T00:23:03.719 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:03 smithi099 ceph-mon[24594]: from='mgr.15333 172.21.15.2:0/842744905' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-17T00:23:03.719 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:03 smithi099 ceph-mon[24594]: from='mgr.15333 ' entity='mgr.y' 2024-09-17T00:23:03.719 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:03 smithi099 ceph-mon[24594]: from='mgr.15333 ' entity='mgr.y' 2024-09-17T00:23:03.719 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:03 smithi099 ceph-mon[24594]: from='mgr.15333 172.21.15.2:0/842744905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:23:03.719 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:03 smithi099 ceph-mon[24594]: from='mgr.15333 172.21.15.2:0/842744905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:23:03.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:03 smithi002 ceph-mon[23737]: from='mgr.15333 ' entity='mgr.y' 2024-09-17T00:23:03.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:03 smithi002 ceph-mon[23737]: from='mgr.15333 ' entity='mgr.y' 2024-09-17T00:23:03.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:03 smithi002 ceph-mon[23737]: from='mgr.15333 ' entity='mgr.y' 2024-09-17T00:23:03.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:03 smithi002 ceph-mon[23737]: from='mgr.15333 ' entity='mgr.y' 2024-09-17T00:23:03.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:03 smithi002 ceph-mon[23737]: mgrmap e32: y(active, since 6s), standbys: x 2024-09-17T00:23:03.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:03 smithi002 ceph-mon[23737]: from='mgr.15333 172.21.15.2:0/842744905' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-17T00:23:03.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:03 smithi002 ceph-mon[23737]: from='mgr.15333 ' entity='mgr.y' 2024-09-17T00:23:03.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:03 smithi002 ceph-mon[23737]: from='mgr.15333 ' entity='mgr.y' 2024-09-17T00:23:03.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:03 smithi002 ceph-mon[23737]: from='mgr.15333 172.21.15.2:0/842744905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:23:03.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:03 smithi002 ceph-mon[23737]: from='mgr.15333 172.21.15.2:0/842744905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:23:03.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:03 smithi002 ceph-mon[28203]: from='mgr.15333 ' entity='mgr.y' 2024-09-17T00:23:03.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:03 smithi002 ceph-mon[28203]: from='mgr.15333 ' entity='mgr.y' 2024-09-17T00:23:03.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:03 smithi002 ceph-mon[28203]: from='mgr.15333 ' entity='mgr.y' 2024-09-17T00:23:03.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:03 smithi002 ceph-mon[28203]: from='mgr.15333 ' entity='mgr.y' 2024-09-17T00:23:03.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:03 smithi002 ceph-mon[28203]: mgrmap e32: y(active, since 6s), standbys: x 2024-09-17T00:23:03.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:03 smithi002 ceph-mon[28203]: from='mgr.15333 172.21.15.2:0/842744905' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-17T00:23:03.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:03 smithi002 ceph-mon[28203]: from='mgr.15333 ' entity='mgr.y' 2024-09-17T00:23:03.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:03 smithi002 ceph-mon[28203]: from='mgr.15333 ' entity='mgr.y' 2024-09-17T00:23:03.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:03 smithi002 ceph-mon[28203]: from='mgr.15333 172.21.15.2:0/842744905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:23:03.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:03 smithi002 ceph-mon[28203]: from='mgr.15333 172.21.15.2:0/842744905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:23:04.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:04 smithi099 ceph-mon[24594]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:23:04.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:04 smithi099 ceph-mon[24594]: Updating smithi002:/etc/ceph/ceph.conf 2024-09-17T00:23:04.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:04 smithi099 ceph-mon[24594]: Updating smithi099:/etc/ceph/ceph.conf 2024-09-17T00:23:04.718 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:04 smithi099 ceph-mon[24594]: Updating smithi002:/var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-17T00:23:04.718 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:04 smithi099 ceph-mon[24594]: Updating smithi099:/var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-17T00:23:04.718 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:04 smithi099 ceph-mon[24594]: from='mgr.15333 ' entity='mgr.y' 2024-09-17T00:23:04.718 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:04 smithi099 ceph-mon[24594]: from='mgr.15333 ' entity='mgr.y' 2024-09-17T00:23:04.718 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:04 smithi099 ceph-mon[24594]: from='mgr.15333 ' entity='mgr.y' 2024-09-17T00:23:04.718 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:04 smithi099 ceph-mon[24594]: from='mgr.15333 ' entity='mgr.y' 2024-09-17T00:23:04.718 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:04 smithi099 ceph-mon[24594]: from='mgr.15333 ' entity='mgr.y' 2024-09-17T00:23:04.718 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:04 smithi099 ceph-mon[24594]: from='mgr.15333 172.21.15.2:0/842744905' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi002.vmouem", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-17T00:23:04.718 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:04 smithi099 ceph-mon[24594]: from='mgr.15333 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi002.vmouem", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-17T00:23:04.718 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:04 smithi099 ceph-mon[24594]: from='mgr.15333 172.21.15.2:0/842744905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:23:04.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:04 smithi002 ceph-mon[23737]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:23:04.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:04 smithi002 ceph-mon[23737]: Updating smithi002:/etc/ceph/ceph.conf 2024-09-17T00:23:04.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:04 smithi002 ceph-mon[23737]: Updating smithi099:/etc/ceph/ceph.conf 2024-09-17T00:23:04.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:04 smithi002 ceph-mon[23737]: Updating smithi002:/var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-17T00:23:04.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:04 smithi002 ceph-mon[23737]: Updating smithi099:/var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-17T00:23:04.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:04 smithi002 ceph-mon[23737]: from='mgr.15333 ' entity='mgr.y' 2024-09-17T00:23:04.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:04 smithi002 ceph-mon[23737]: from='mgr.15333 ' entity='mgr.y' 2024-09-17T00:23:04.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:04 smithi002 ceph-mon[23737]: from='mgr.15333 ' entity='mgr.y' 2024-09-17T00:23:04.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:04 smithi002 ceph-mon[23737]: from='mgr.15333 ' entity='mgr.y' 2024-09-17T00:23:04.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:04 smithi002 ceph-mon[23737]: from='mgr.15333 ' entity='mgr.y' 2024-09-17T00:23:04.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:04 smithi002 ceph-mon[23737]: from='mgr.15333 172.21.15.2:0/842744905' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi002.vmouem", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-17T00:23:04.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:04 smithi002 ceph-mon[23737]: from='mgr.15333 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi002.vmouem", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-17T00:23:04.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:04 smithi002 ceph-mon[23737]: from='mgr.15333 172.21.15.2:0/842744905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:23:04.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:04 smithi002 ceph-mon[28203]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:23:04.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:04 smithi002 ceph-mon[28203]: Updating smithi002:/etc/ceph/ceph.conf 2024-09-17T00:23:04.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:04 smithi002 ceph-mon[28203]: Updating smithi099:/etc/ceph/ceph.conf 2024-09-17T00:23:04.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:04 smithi002 ceph-mon[28203]: Updating smithi002:/var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-17T00:23:04.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:04 smithi002 ceph-mon[28203]: Updating smithi099:/var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-17T00:23:04.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:04 smithi002 ceph-mon[28203]: from='mgr.15333 ' entity='mgr.y' 2024-09-17T00:23:04.731 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:04 smithi002 ceph-mon[28203]: from='mgr.15333 ' entity='mgr.y' 2024-09-17T00:23:04.731 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:04 smithi002 ceph-mon[28203]: from='mgr.15333 ' entity='mgr.y' 2024-09-17T00:23:04.731 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:04 smithi002 ceph-mon[28203]: from='mgr.15333 ' entity='mgr.y' 2024-09-17T00:23:04.731 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:04 smithi002 ceph-mon[28203]: from='mgr.15333 ' entity='mgr.y' 2024-09-17T00:23:04.731 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:04 smithi002 ceph-mon[28203]: from='mgr.15333 172.21.15.2:0/842744905' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi002.vmouem", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-17T00:23:04.731 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:04 smithi002 ceph-mon[28203]: from='mgr.15333 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi002.vmouem", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-17T00:23:04.731 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:04 smithi002 ceph-mon[28203]: from='mgr.15333 172.21.15.2:0/842744905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:23:05.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:05 smithi099 ceph-mon[24594]: Updating smithi002:/etc/ceph/ceph.client.admin.keyring 2024-09-17T00:23:05.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:05 smithi099 ceph-mon[24594]: Updating smithi099:/etc/ceph/ceph.client.admin.keyring 2024-09-17T00:23:05.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:05 smithi099 ceph-mon[24594]: Updating smithi002:/var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-17T00:23:05.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:05 smithi099 ceph-mon[24594]: Updating smithi099:/var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-17T00:23:05.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:05 smithi099 ceph-mon[24594]: Reconfiguring iscsi.foo.smithi002.vmouem (dependencies changed)... 2024-09-17T00:23:05.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:05 smithi099 ceph-mon[24594]: Reconfiguring daemon iscsi.foo.smithi002.vmouem on smithi002 2024-09-17T00:23:05.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:05 smithi002 ceph-mon[28203]: Updating smithi002:/etc/ceph/ceph.client.admin.keyring 2024-09-17T00:23:05.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:05 smithi002 ceph-mon[28203]: Updating smithi099:/etc/ceph/ceph.client.admin.keyring 2024-09-17T00:23:05.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:05 smithi002 ceph-mon[28203]: Updating smithi002:/var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-17T00:23:05.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:05 smithi002 ceph-mon[28203]: Updating smithi099:/var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-17T00:23:05.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:05 smithi002 ceph-mon[28203]: Reconfiguring iscsi.foo.smithi002.vmouem (dependencies changed)... 2024-09-17T00:23:05.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:05 smithi002 ceph-mon[28203]: Reconfiguring daemon iscsi.foo.smithi002.vmouem on smithi002 2024-09-17T00:23:05.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:05 smithi002 ceph-mon[23737]: Updating smithi002:/etc/ceph/ceph.client.admin.keyring 2024-09-17T00:23:05.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:05 smithi002 ceph-mon[23737]: Updating smithi099:/etc/ceph/ceph.client.admin.keyring 2024-09-17T00:23:05.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:05 smithi002 ceph-mon[23737]: Updating smithi002:/var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-17T00:23:05.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:05 smithi002 ceph-mon[23737]: Updating smithi099:/var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-17T00:23:05.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:05 smithi002 ceph-mon[23737]: Reconfiguring iscsi.foo.smithi002.vmouem (dependencies changed)... 2024-09-17T00:23:05.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:05 smithi002 ceph-mon[23737]: Reconfiguring daemon iscsi.foo.smithi002.vmouem on smithi002 2024-09-17T00:23:06.635 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:06 smithi002 ceph-mon[23737]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-09-17T00:23:06.636 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:06 smithi002 ceph-mon[28203]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-09-17T00:23:06.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:06 smithi099 ceph-mon[24594]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-09-17T00:23:07.049 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:23:06 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[45656]: ::ffff:172.21.15.99 - - [17/Sep/2024:00:23:06] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-17T00:23:07.468 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:07 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: 2024-09-17T00:23:07.264+0000 7f89aca67640 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi002.vmouem ... 2024-09-17T00:23:07.469 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:07 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: Creating ceph-iscsi config... 2024-09-17T00:23:07.469 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:07 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/iscsi-gateway.cfg 2024-09-17T00:23:07.469 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:07 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/tcmu-runner-entrypoint.sh 2024-09-17T00:23:07.469 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:07 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: Non-zero exit code 1 from systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem 2024-09-17T00:23:07.469 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:07 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: systemctl: stderr Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:23:07.469 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:07 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: systemctl: stderr See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:23:07.469 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:07 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: Traceback (most recent call last): 2024-09-17T00:23:07.469 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:07 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-17T00:23:07.469 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:07 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: return _run_code(code, main_globals, None, 2024-09-17T00:23:07.469 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:07 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-17T00:23:07.469 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:07 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: exec(code, run_globals) 2024-09-17T00:23:07.469 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:07 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5579, in 2024-09-17T00:23:07.469 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:07 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5567, in main 2024-09-17T00:23:07.470 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:07 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3049, in command_deploy_from 2024-09-17T00:23:07.470 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:07 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3084, in _common_deploy 2024-09-17T00:23:07.470 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:07 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3104, in _deploy_daemon_container 2024-09-17T00:23:07.470 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:07 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 1124, in deploy_daemon 2024-09-17T00:23:07.470 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:07 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/tmp/tmpnroy8d0w.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-17T00:23:07.470 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:07 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: RuntimeError: Failed command: systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem: Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:23:07.470 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:07 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:23:07.470 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:07 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: Traceback (most recent call last): 2024-09-17T00:23:07.471 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:07 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1134, in _check_daemons 2024-09-17T00:23:07.471 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:07 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: self.mgr._daemon_action(daemon_spec, action=action) 2024-09-17T00:23:07.471 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:07 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2473, in _daemon_action 2024-09-17T00:23:07.471 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:07 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: return self.wait_async( 2024-09-17T00:23:07.471 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:07 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/usr/share/ceph/mgr/cephadm/module.py", line 796, in wait_async 2024-09-17T00:23:07.471 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:07 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: return self.event_loop.get_result(coro, timeout) 2024-09-17T00:23:07.471 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:07 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-09-17T00:23:07.471 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:07 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: return future.result(timeout) 2024-09-17T00:23:07.471 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:07 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/lib64/python3.9/concurrent/futures/_base.py", line 446, in result 2024-09-17T00:23:07.471 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:07 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: return self.__get_result() 2024-09-17T00:23:07.471 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:07 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/lib64/python3.9/concurrent/futures/_base.py", line 391, in __get_result 2024-09-17T00:23:07.471 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:07 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: raise self._exception 2024-09-17T00:23:07.471 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:07 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1370, in _create_daemon 2024-09-17T00:23:07.471 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:07 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: out, err, code = await self._run_cephadm( 2024-09-17T00:23:07.472 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:07 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1713, in _run_cephadm 2024-09-17T00:23:07.472 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:07 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: raise OrchestratorError( 2024-09-17T00:23:07.472 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:07 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi002.vmouem ... 2024-09-17T00:23:07.472 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:07 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: Creating ceph-iscsi config... 2024-09-17T00:23:07.472 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:07 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/iscsi-gateway.cfg 2024-09-17T00:23:07.472 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:07 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/tcmu-runner-entrypoint.sh 2024-09-17T00:23:07.472 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:07 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: Non-zero exit code 1 from systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem 2024-09-17T00:23:07.472 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:07 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: systemctl: stderr Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:23:07.472 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:07 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: systemctl: stderr See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:23:07.472 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:07 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: Traceback (most recent call last): 2024-09-17T00:23:07.472 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:07 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-17T00:23:07.472 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:07 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: return _run_code(code, main_globals, None, 2024-09-17T00:23:07.472 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:07 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-17T00:23:07.472 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:07 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: exec(code, run_globals) 2024-09-17T00:23:07.473 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:07 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5579, in 2024-09-17T00:23:07.473 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:07 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5567, in main 2024-09-17T00:23:07.473 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:07 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3049, in command_deploy_from 2024-09-17T00:23:07.473 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:07 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3084, in _common_deploy 2024-09-17T00:23:07.473 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:07 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3104, in _deploy_daemon_container 2024-09-17T00:23:07.473 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:07 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 1124, in deploy_daemon 2024-09-17T00:23:07.473 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:07 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/tmp/tmpnroy8d0w.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-17T00:23:07.473 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:07 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: RuntimeError: Failed command: systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem: Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:23:07.473 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:07 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:23:08.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:08 smithi099 ceph-mon[24594]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 0 B/s wr, 13 op/s 2024-09-17T00:23:08.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:08 smithi099 ceph-mon[24594]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi002.vmouem ... 2024-09-17T00:23:08.467 INFO:journalctl@ceph.mon.b.smithi099.stdout: Creating ceph-iscsi config... 2024-09-17T00:23:08.467 INFO:journalctl@ceph.mon.b.smithi099.stdout: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/iscsi-gateway.cfg 2024-09-17T00:23:08.467 INFO:journalctl@ceph.mon.b.smithi099.stdout: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/tcmu-runner-entrypoint.sh 2024-09-17T00:23:08.468 INFO:journalctl@ceph.mon.b.smithi099.stdout: Non-zero exit code 1 from systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem 2024-09-17T00:23:08.468 INFO:journalctl@ceph.mon.b.smithi099.stdout: systemctl: stderr Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:23:08.468 INFO:journalctl@ceph.mon.b.smithi099.stdout: systemctl: stderr See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:23:08.468 INFO:journalctl@ceph.mon.b.smithi099.stdout: Traceback (most recent call last): 2024-09-17T00:23:08.468 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-17T00:23:08.468 INFO:journalctl@ceph.mon.b.smithi099.stdout: return _run_code(code, main_globals, None, 2024-09-17T00:23:08.468 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-17T00:23:08.468 INFO:journalctl@ceph.mon.b.smithi099.stdout: exec(code, run_globals) 2024-09-17T00:23:08.468 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5579, in 2024-09-17T00:23:08.468 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5567, in main 2024-09-17T00:23:08.468 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3049, in command_deploy_from 2024-09-17T00:23:08.468 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3084, in _common_deploy 2024-09-17T00:23:08.469 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3104, in _deploy_daemon_container 2024-09-17T00:23:08.469 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 1124, in deploy_daemon 2024-09-17T00:23:08.469 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-17T00:23:08.469 INFO:journalctl@ceph.mon.b.smithi099.stdout: RuntimeError: Failed command: systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem: Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:23:08.469 INFO:journalctl@ceph.mon.b.smithi099.stdout: See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:23:08.469 INFO:journalctl@ceph.mon.b.smithi099.stdout: Traceback (most recent call last): 2024-09-17T00:23:08.469 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1134, in _check_daemons 2024-09-17T00:23:08.469 INFO:journalctl@ceph.mon.b.smithi099.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-09-17T00:23:08.469 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2473, in _daemon_action 2024-09-17T00:23:08.469 INFO:journalctl@ceph.mon.b.smithi099.stdout: return self.wait_async( 2024-09-17T00:23:08.469 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 796, in wait_async 2024-09-17T00:23:08.469 INFO:journalctl@ceph.mon.b.smithi099.stdout: return self.event_loop.get_result(coro, timeout) 2024-09-17T00:23:08.469 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-09-17T00:23:08.470 INFO:journalctl@ceph.mon.b.smithi099.stdout: return future.result(timeout) 2024-09-17T00:23:08.470 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/lib64/python3.9/concurrent/futures/_base.py", line 446, in result 2024-09-17T00:23:08.470 INFO:journalctl@ceph.mon.b.smithi099.stdout: return self.__get_result() 2024-09-17T00:23:08.470 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/lib64/python3.9/concurrent/futures/_base.py", line 391, in __get_result 2024-09-17T00:23:08.470 INFO:journalctl@ceph.mon.b.smithi099.stdout: raise self._exception 2024-09-17T00:23:08.470 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1370, in _create_daemon 2024-09-17T00:23:08.470 INFO:journalctl@ceph.mon.b.smithi099.stdout: out, err, code = await self._run_cephadm( 2024-09-17T00:23:08.470 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1713, in _run_cephadm 2024-09-17T00:23:08.470 INFO:journalctl@ceph.mon.b.smithi099.stdout: raise OrchestratorError( 2024-09-17T00:23:08.470 INFO:journalctl@ceph.mon.b.smithi099.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi002.vmouem ... 2024-09-17T00:23:08.470 INFO:journalctl@ceph.mon.b.smithi099.stdout: Creating ceph-iscsi config... 2024-09-17T00:23:08.470 INFO:journalctl@ceph.mon.b.smithi099.stdout: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/iscsi-gateway.cfg 2024-09-17T00:23:08.470 INFO:journalctl@ceph.mon.b.smithi099.stdout: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/tcmu-runner-entrypoint.sh 2024-09-17T00:23:08.470 INFO:journalctl@ceph.mon.b.smithi099.stdout: Non-zero exit code 1 from systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem 2024-09-17T00:23:08.471 INFO:journalctl@ceph.mon.b.smithi099.stdout: systemctl: stderr Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:23:08.471 INFO:journalctl@ceph.mon.b.smithi099.stdout: systemctl: stderr See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:23:08.471 INFO:journalctl@ceph.mon.b.smithi099.stdout: Traceback (most recent call last): 2024-09-17T00:23:08.471 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-17T00:23:08.471 INFO:journalctl@ceph.mon.b.smithi099.stdout: return _run_code(code, main_globals, None, 2024-09-17T00:23:08.471 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-17T00:23:08.471 INFO:journalctl@ceph.mon.b.smithi099.stdout: exec(code, run_globals) 2024-09-17T00:23:08.471 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5579, in 2024-09-17T00:23:08.471 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5567, in main 2024-09-17T00:23:08.471 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3049, in command_deploy_from 2024-09-17T00:23:08.471 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3084, in _common_deploy 2024-09-17T00:23:08.471 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3104, in _deploy_daemon_container 2024-09-17T00:23:08.471 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 1124, in deploy_daemon 2024-09-17T00:23:08.471 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-17T00:23:08.472 INFO:journalctl@ceph.mon.b.smithi099.stdout: RuntimeError: Failed command: systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem: Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:23:08.472 INFO:journalctl@ceph.mon.b.smithi099.stdout: See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:23:08.472 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:08 smithi099 ceph-mon[24594]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-17T00:23:08.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:08 smithi002 ceph-mon[23737]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 0 B/s wr, 13 op/s 2024-09-17T00:23:08.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:08 smithi002 ceph-mon[23737]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi002.vmouem ... 2024-09-17T00:23:08.730 INFO:journalctl@ceph.mon.a.smithi002.stdout: Creating ceph-iscsi config... 2024-09-17T00:23:08.730 INFO:journalctl@ceph.mon.a.smithi002.stdout: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/iscsi-gateway.cfg 2024-09-17T00:23:08.730 INFO:journalctl@ceph.mon.a.smithi002.stdout: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/tcmu-runner-entrypoint.sh 2024-09-17T00:23:08.730 INFO:journalctl@ceph.mon.a.smithi002.stdout: Non-zero exit code 1 from systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem 2024-09-17T00:23:08.730 INFO:journalctl@ceph.mon.a.smithi002.stdout: systemctl: stderr Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:23:08.730 INFO:journalctl@ceph.mon.a.smithi002.stdout: systemctl: stderr See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:23:08.730 INFO:journalctl@ceph.mon.a.smithi002.stdout: Traceback (most recent call last): 2024-09-17T00:23:08.730 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-17T00:23:08.731 INFO:journalctl@ceph.mon.a.smithi002.stdout: return _run_code(code, main_globals, None, 2024-09-17T00:23:08.731 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-17T00:23:08.731 INFO:journalctl@ceph.mon.a.smithi002.stdout: exec(code, run_globals) 2024-09-17T00:23:08.731 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5579, in 2024-09-17T00:23:08.731 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5567, in main 2024-09-17T00:23:08.731 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3049, in command_deploy_from 2024-09-17T00:23:08.731 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3084, in _common_deploy 2024-09-17T00:23:08.731 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3104, in _deploy_daemon_container 2024-09-17T00:23:08.731 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 1124, in deploy_daemon 2024-09-17T00:23:08.731 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-17T00:23:08.731 INFO:journalctl@ceph.mon.a.smithi002.stdout: RuntimeError: Failed command: systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem: Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:23:08.732 INFO:journalctl@ceph.mon.a.smithi002.stdout: See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:23:08.732 INFO:journalctl@ceph.mon.a.smithi002.stdout: Traceback (most recent call last): 2024-09-17T00:23:08.732 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1134, in _check_daemons 2024-09-17T00:23:08.732 INFO:journalctl@ceph.mon.a.smithi002.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-09-17T00:23:08.732 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2473, in _daemon_action 2024-09-17T00:23:08.732 INFO:journalctl@ceph.mon.a.smithi002.stdout: return self.wait_async( 2024-09-17T00:23:08.732 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 796, in wait_async 2024-09-17T00:23:08.732 INFO:journalctl@ceph.mon.a.smithi002.stdout: return self.event_loop.get_result(coro, timeout) 2024-09-17T00:23:08.732 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-09-17T00:23:08.733 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:08 smithi002 ceph-mon[28203]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 0 B/s wr, 13 op/s 2024-09-17T00:23:08.733 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:08 smithi002 ceph-mon[28203]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi002.vmouem ... 2024-09-17T00:23:08.733 INFO:journalctl@ceph.mon.c.smithi002.stdout: Creating ceph-iscsi config... 2024-09-17T00:23:08.733 INFO:journalctl@ceph.mon.c.smithi002.stdout: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/iscsi-gateway.cfg 2024-09-17T00:23:08.733 INFO:journalctl@ceph.mon.c.smithi002.stdout: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/tcmu-runner-entrypoint.sh 2024-09-17T00:23:08.733 INFO:journalctl@ceph.mon.c.smithi002.stdout: Non-zero exit code 1 from systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem 2024-09-17T00:23:08.733 INFO:journalctl@ceph.mon.c.smithi002.stdout: systemctl: stderr Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:23:08.733 INFO:journalctl@ceph.mon.c.smithi002.stdout: systemctl: stderr See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:23:08.733 INFO:journalctl@ceph.mon.c.smithi002.stdout: Traceback (most recent call last): 2024-09-17T00:23:08.733 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-17T00:23:08.733 INFO:journalctl@ceph.mon.c.smithi002.stdout: return _run_code(code, main_globals, None, 2024-09-17T00:23:08.733 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-17T00:23:08.733 INFO:journalctl@ceph.mon.c.smithi002.stdout: exec(code, run_globals) 2024-09-17T00:23:08.733 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5579, in 2024-09-17T00:23:08.734 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5567, in main 2024-09-17T00:23:08.734 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3049, in command_deploy_from 2024-09-17T00:23:08.734 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3084, in _common_deploy 2024-09-17T00:23:08.734 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3104, in _deploy_daemon_container 2024-09-17T00:23:08.734 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 1124, in deploy_daemon 2024-09-17T00:23:08.734 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-17T00:23:08.734 INFO:journalctl@ceph.mon.c.smithi002.stdout: RuntimeError: Failed command: systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem: Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:23:08.734 INFO:journalctl@ceph.mon.c.smithi002.stdout: See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:23:08.734 INFO:journalctl@ceph.mon.c.smithi002.stdout: Traceback (most recent call last): 2024-09-17T00:23:08.734 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1134, in _check_daemons 2024-09-17T00:23:08.734 INFO:journalctl@ceph.mon.c.smithi002.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-09-17T00:23:08.734 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2473, in _daemon_action 2024-09-17T00:23:08.734 INFO:journalctl@ceph.mon.c.smithi002.stdout: return self.wait_async( 2024-09-17T00:23:08.735 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 796, in wait_async 2024-09-17T00:23:08.735 INFO:journalctl@ceph.mon.c.smithi002.stdout: return self.event_loop.get_result(coro, timeout) 2024-09-17T00:23:08.735 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-09-17T00:23:08.735 INFO:journalctl@ceph.mon.a.smithi002.stdout: return future.result(timeout) 2024-09-17T00:23:08.735 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/lib64/python3.9/concurrent/futures/_base.py", line 446, in result 2024-09-17T00:23:08.735 INFO:journalctl@ceph.mon.a.smithi002.stdout: return self.__get_result() 2024-09-17T00:23:08.735 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/lib64/python3.9/concurrent/futures/_base.py", line 391, in __get_result 2024-09-17T00:23:08.735 INFO:journalctl@ceph.mon.a.smithi002.stdout: raise self._exception 2024-09-17T00:23:08.735 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1370, in _create_daemon 2024-09-17T00:23:08.735 INFO:journalctl@ceph.mon.a.smithi002.stdout: out, err, code = await self._run_cephadm( 2024-09-17T00:23:08.735 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1713, in _run_cephadm 2024-09-17T00:23:08.736 INFO:journalctl@ceph.mon.a.smithi002.stdout: raise OrchestratorError( 2024-09-17T00:23:08.736 INFO:journalctl@ceph.mon.a.smithi002.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi002.vmouem ... 2024-09-17T00:23:08.736 INFO:journalctl@ceph.mon.a.smithi002.stdout: Creating ceph-iscsi config... 2024-09-17T00:23:08.736 INFO:journalctl@ceph.mon.a.smithi002.stdout: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/iscsi-gateway.cfg 2024-09-17T00:23:08.736 INFO:journalctl@ceph.mon.a.smithi002.stdout: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/tcmu-runner-entrypoint.sh 2024-09-17T00:23:08.736 INFO:journalctl@ceph.mon.a.smithi002.stdout: Non-zero exit code 1 from systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem 2024-09-17T00:23:08.736 INFO:journalctl@ceph.mon.a.smithi002.stdout: systemctl: stderr Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:23:08.736 INFO:journalctl@ceph.mon.a.smithi002.stdout: systemctl: stderr See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:23:08.736 INFO:journalctl@ceph.mon.a.smithi002.stdout: Traceback (most recent call last): 2024-09-17T00:23:08.736 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-17T00:23:08.736 INFO:journalctl@ceph.mon.a.smithi002.stdout: return _run_code(code, main_globals, None, 2024-09-17T00:23:08.736 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-17T00:23:08.736 INFO:journalctl@ceph.mon.a.smithi002.stdout: exec(code, run_globals) 2024-09-17T00:23:08.737 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5579, in 2024-09-17T00:23:08.737 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5567, in main 2024-09-17T00:23:08.737 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3049, in command_deploy_from 2024-09-17T00:23:08.737 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3084, in _common_deploy 2024-09-17T00:23:08.737 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3104, in _deploy_daemon_container 2024-09-17T00:23:08.737 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 1124, in deploy_daemon 2024-09-17T00:23:08.737 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-17T00:23:08.737 INFO:journalctl@ceph.mon.a.smithi002.stdout: RuntimeError: Failed command: systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem: Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:23:08.737 INFO:journalctl@ceph.mon.a.smithi002.stdout: See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:23:08.737 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:08 smithi002 ceph-mon[23737]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-17T00:23:08.737 INFO:journalctl@ceph.mon.c.smithi002.stdout: return future.result(timeout) 2024-09-17T00:23:08.737 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/lib64/python3.9/concurrent/futures/_base.py", line 446, in result 2024-09-17T00:23:08.738 INFO:journalctl@ceph.mon.c.smithi002.stdout: return self.__get_result() 2024-09-17T00:23:08.738 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/lib64/python3.9/concurrent/futures/_base.py", line 391, in __get_result 2024-09-17T00:23:08.738 INFO:journalctl@ceph.mon.c.smithi002.stdout: raise self._exception 2024-09-17T00:23:08.738 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1370, in _create_daemon 2024-09-17T00:23:08.738 INFO:journalctl@ceph.mon.c.smithi002.stdout: out, err, code = await self._run_cephadm( 2024-09-17T00:23:08.738 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1713, in _run_cephadm 2024-09-17T00:23:08.738 INFO:journalctl@ceph.mon.c.smithi002.stdout: raise OrchestratorError( 2024-09-17T00:23:08.738 INFO:journalctl@ceph.mon.c.smithi002.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi002.vmouem ... 2024-09-17T00:23:08.738 INFO:journalctl@ceph.mon.c.smithi002.stdout: Creating ceph-iscsi config... 2024-09-17T00:23:08.738 INFO:journalctl@ceph.mon.c.smithi002.stdout: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/iscsi-gateway.cfg 2024-09-17T00:23:08.738 INFO:journalctl@ceph.mon.c.smithi002.stdout: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/tcmu-runner-entrypoint.sh 2024-09-17T00:23:08.738 INFO:journalctl@ceph.mon.c.smithi002.stdout: Non-zero exit code 1 from systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem 2024-09-17T00:23:08.738 INFO:journalctl@ceph.mon.c.smithi002.stdout: systemctl: stderr Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:23:08.738 INFO:journalctl@ceph.mon.c.smithi002.stdout: systemctl: stderr See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:23:08.739 INFO:journalctl@ceph.mon.c.smithi002.stdout: Traceback (most recent call last): 2024-09-17T00:23:08.739 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-17T00:23:08.739 INFO:journalctl@ceph.mon.c.smithi002.stdout: return _run_code(code, main_globals, None, 2024-09-17T00:23:08.739 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-17T00:23:08.739 INFO:journalctl@ceph.mon.c.smithi002.stdout: exec(code, run_globals) 2024-09-17T00:23:08.739 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5579, in 2024-09-17T00:23:08.739 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5567, in main 2024-09-17T00:23:08.739 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3049, in command_deploy_from 2024-09-17T00:23:08.739 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3084, in _common_deploy 2024-09-17T00:23:08.739 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3104, in _deploy_daemon_container 2024-09-17T00:23:08.739 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 1124, in deploy_daemon 2024-09-17T00:23:08.739 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-17T00:23:08.739 INFO:journalctl@ceph.mon.c.smithi002.stdout: RuntimeError: Failed command: systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem: Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:23:08.739 INFO:journalctl@ceph.mon.c.smithi002.stdout: See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:23:08.740 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:08 smithi002 ceph-mon[28203]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-17T00:23:09.651 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:09 smithi099 ceph-mon[24594]: Reconfiguring daemon prometheus.a on smithi099 2024-09-17T00:23:09.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:09 smithi002 ceph-mon[23737]: Reconfiguring daemon prometheus.a on smithi099 2024-09-17T00:23:09.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:09 smithi002 ceph-mon[28203]: Reconfiguring daemon prometheus.a on smithi099 2024-09-17T00:23:10.143 INFO:teuthology.orchestra.run.smithi002.stdout:true 2024-09-17T00:23:10.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:10 smithi099 ceph-mon[24594]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-17T00:23:10.717 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:23:10 smithi099 systemd[1]: Stopping Ceph prometheus.a for 3aad27a2-7489-11ef-bceb-c7b262605968... 2024-09-17T00:23:10.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:10 smithi002 ceph-mon[28203]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-17T00:23:10.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:10 smithi002 ceph-mon[23737]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-17T00:23:10.822 INFO:teuthology.orchestra.run.smithi002.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-17T00:23:10.822 INFO:teuthology.orchestra.run.smithi002.stdout:alertmanager.a smithi002 *:9093,9094 running (60s) 8s ago 5m 16.9M - 0.25.0 c8568f914cd2 a21e13b3d677 2024-09-17T00:23:10.823 INFO:teuthology.orchestra.run.smithi002.stdout:grafana.a smithi099 *:3000 running (34s) 9s ago 5m 43.0M - 8.3.5 dad864ee21e9 bafb824a3c08 2024-09-17T00:23:10.823 INFO:teuthology.orchestra.run.smithi002.stdout:iscsi.foo.smithi002.vmouem smithi002 running (15s) 8s ago 4m 45.9M - 3.5 e1d6a67b021e cdd8a2092ce4 2024-09-17T00:23:10.823 INFO:teuthology.orchestra.run.smithi002.stdout:mgr.x smithi099 *:8443,9283 running (95s) 9s ago 10m 96.6M - 19.1.1-330-gc2ddf972 ebb0615149de a8e1e8964c84 2024-09-17T00:23:10.823 INFO:teuthology.orchestra.run.smithi002.stdout:mgr.y smithi002 *:8443,9283,8765 running (28s) 8s ago 12m 474M - 19.1.1-330-gc2ddf972 ebb0615149de 8838897e8449 2024-09-17T00:23:10.823 INFO:teuthology.orchestra.run.smithi002.stdout:mon.a smithi002 running (12m) 8s ago 12m 58.1M 2048M 17.2.0 e1d6a67b021e a68823df625f 2024-09-17T00:23:10.823 INFO:teuthology.orchestra.run.smithi002.stdout:mon.b smithi099 running (10m) 9s ago 10m 45.6M 2048M 17.2.0 e1d6a67b021e c6d2af715187 2024-09-17T00:23:10.823 INFO:teuthology.orchestra.run.smithi002.stdout:mon.c smithi002 running (10m) 8s ago 10m 43.3M 2048M 17.2.0 e1d6a67b021e 77b3d2e14899 2024-09-17T00:23:10.823 INFO:teuthology.orchestra.run.smithi002.stdout:node-exporter.a smithi002 *:9100 running (70s) 8s ago 6m 8002k - 1.5.0 0da6a335fe13 1030ae01bb32 2024-09-17T00:23:10.823 INFO:teuthology.orchestra.run.smithi002.stdout:node-exporter.b smithi099 *:9100 running (38s) 9s ago 6m 8400k - 1.5.0 0da6a335fe13 01fcd35dca8a 2024-09-17T00:23:10.823 INFO:teuthology.orchestra.run.smithi002.stdout:osd.0 smithi002 running (10m) 8s ago 10m 52.4M 2737M 17.2.0 e1d6a67b021e a0ad0fb7aa34 2024-09-17T00:23:10.823 INFO:teuthology.orchestra.run.smithi002.stdout:osd.1 smithi002 running (9m) 8s ago 9m 55.5M 2737M 17.2.0 e1d6a67b021e 7cb62c311c7d 2024-09-17T00:23:10.823 INFO:teuthology.orchestra.run.smithi002.stdout:osd.2 smithi002 running (9m) 8s ago 9m 51.8M 2737M 17.2.0 e1d6a67b021e eafddacb955e 2024-09-17T00:23:10.823 INFO:teuthology.orchestra.run.smithi002.stdout:osd.3 smithi002 running (8m) 8s ago 8m 55.8M 2737M 17.2.0 e1d6a67b021e fd99d3764412 2024-09-17T00:23:10.824 INFO:teuthology.orchestra.run.smithi002.stdout:osd.4 smithi099 running (8m) 9s ago 8m 55.5M 2992M 17.2.0 e1d6a67b021e 0af1827b658d 2024-09-17T00:23:10.824 INFO:teuthology.orchestra.run.smithi002.stdout:osd.5 smithi099 running (8m) 9s ago 8m 54.2M 2992M 17.2.0 e1d6a67b021e 4f7ef3aefc58 2024-09-17T00:23:10.824 INFO:teuthology.orchestra.run.smithi002.stdout:osd.6 smithi099 running (7m) 9s ago 7m 49.6M 2992M 17.2.0 e1d6a67b021e 3213f5e6952a 2024-09-17T00:23:10.824 INFO:teuthology.orchestra.run.smithi002.stdout:osd.7 smithi099 running (7m) 9s ago 7m 56.2M 2992M 17.2.0 e1d6a67b021e 63be3a1ed0a1 2024-09-17T00:23:10.824 INFO:teuthology.orchestra.run.smithi002.stdout:prometheus.a smithi099 *:9095 running (44s) 9s ago 6m 44.8M - 2.43.0 a07b618ecd1d 1d14ff0e495e 2024-09-17T00:23:10.824 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.foo.smithi002.uwdhcs smithi002 *:8000 running (5m) 8s ago 5m 74.1M - 17.2.0 e1d6a67b021e 9c35813f459a 2024-09-17T00:23:10.824 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.foo.smithi099.utzfap smithi099 *:8000 running (5m) 9s ago 5m 74.6M - 17.2.0 e1d6a67b021e 34ec989160a6 2024-09-17T00:23:10.824 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.smpl.smithi002.rqnxii smithi002 *:80 running (4m) 8s ago 4m 72.6M - 17.2.0 e1d6a67b021e 0e180d0afb99 2024-09-17T00:23:10.824 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.smpl.smithi099.nzvkpm smithi099 *:80 running (4m) 9s ago 4m 75.3M - 17.2.0 e1d6a67b021e 6a068e82ac72 2024-09-17T00:23:11.109 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:23:10 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[49368]: ts=2024-09-17T00:23:10.820Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-09-17T00:23:11.109 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:23:10 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[49368]: ts=2024-09-17T00:23:10.820Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-09-17T00:23:11.109 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:23:10 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[49368]: ts=2024-09-17T00:23:10.820Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-09-17T00:23:11.109 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:23:10 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[49368]: ts=2024-09-17T00:23:10.820Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-09-17T00:23:11.110 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:23:10 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[49368]: ts=2024-09-17T00:23:10.820Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-09-17T00:23:11.110 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:23:10 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[49368]: ts=2024-09-17T00:23:10.820Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-09-17T00:23:11.110 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:23:10 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[49368]: ts=2024-09-17T00:23:10.820Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-09-17T00:23:11.110 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:23:10 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[49368]: ts=2024-09-17T00:23:10.820Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-09-17T00:23:11.110 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:23:10 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[49368]: ts=2024-09-17T00:23:10.821Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-09-17T00:23:11.110 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:23:10 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[49368]: ts=2024-09-17T00:23:10.821Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-09-17T00:23:11.110 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:23:10 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[49368]: ts=2024-09-17T00:23:10.821Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-09-17T00:23:11.110 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:23:10 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[49368]: ts=2024-09-17T00:23:10.821Z caller=main.go:1147 level=info msg="See you next time!" 2024-09-17T00:23:11.110 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:23:10 smithi099 podman[52634]: 2024-09-17 00:23:10.83095739 +0000 UTC m=+0.135464049 container died 1d14ff0e495ea18aeb3efae630ad2af8b255b51b51b50169ab3e0077fac846d2 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-17T00:23:11.233 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-09-17T00:23:11.233 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": { 2024-09-17T00:23:11.233 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-09-17T00:23:11.233 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-09-17T00:23:11.233 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": { 2024-09-17T00:23:11.233 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 19.1.1-330-gc2ddf972 (c2ddf9721ef254645201cdbabd9c8f49c30a586f) squid (stable)": 2 2024-09-17T00:23:11.233 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-09-17T00:23:11.233 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": { 2024-09-17T00:23:11.233 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-09-17T00:23:11.233 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-09-17T00:23:11.234 INFO:teuthology.orchestra.run.smithi002.stdout: "mds": {}, 2024-09-17T00:23:11.234 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw": { 2024-09-17T00:23:11.234 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-09-17T00:23:11.234 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-09-17T00:23:11.234 INFO:teuthology.orchestra.run.smithi002.stdout: "overall": { 2024-09-17T00:23:11.234 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 15, 2024-09-17T00:23:11.234 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 19.1.1-330-gc2ddf972 (c2ddf9721ef254645201cdbabd9c8f49c30a586f) squid (stable)": 2 2024-09-17T00:23:11.234 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-09-17T00:23:11.234 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-09-17T00:23:11.443 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:23:11 smithi099 podman[52634]: 2024-09-17 00:23:11.21210624 +0000 UTC m=+0.516612895 container cleanup 1d14ff0e495ea18aeb3efae630ad2af8b255b51b51b50169ab3e0077fac846d2 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-17T00:23:11.443 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:23:11 smithi099 bash[52634]: ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a 2024-09-17T00:23:11.584 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-09-17T00:23:11.584 INFO:teuthology.orchestra.run.smithi002.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f", 2024-09-17T00:23:11.584 INFO:teuthology.orchestra.run.smithi002.stdout: "in_progress": true, 2024-09-17T00:23:11.584 INFO:teuthology.orchestra.run.smithi002.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-17T00:23:11.584 INFO:teuthology.orchestra.run.smithi002.stdout: "services_complete": [ 2024-09-17T00:23:11.584 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr" 2024-09-17T00:23:11.584 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-09-17T00:23:11.585 INFO:teuthology.orchestra.run.smithi002.stdout: "progress": "2/23 daemons upgraded", 2024-09-17T00:23:11.585 INFO:teuthology.orchestra.run.smithi002.stdout: "message": "", 2024-09-17T00:23:11.585 INFO:teuthology.orchestra.run.smithi002.stdout: "is_paused": false 2024-09-17T00:23:11.585 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-09-17T00:23:11.585 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:11 smithi002 ceph-mon[28203]: from='client.15393 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:23:11.585 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:11 smithi002 ceph-mon[28203]: from='client.25285 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:23:11.585 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:11 smithi002 ceph-mon[28203]: from='client.? 172.21.15.2:0/224225078' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:23:11.585 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:11 smithi002 ceph-mon[23737]: from='client.15393 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:23:11.586 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:11 smithi002 ceph-mon[23737]: from='client.25285 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:23:11.586 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:11 smithi002 ceph-mon[23737]: from='client.? 172.21.15.2:0/224225078' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:23:11.716 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:11 smithi099 ceph-mon[24594]: from='client.15393 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:23:11.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:11 smithi099 ceph-mon[24594]: from='client.25285 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:23:11.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:11 smithi099 ceph-mon[24594]: from='client.? 172.21.15.2:0/224225078' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:23:11.717 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:23:11 smithi099 podman[52646]: 2024-09-17 00:23:11.463108186 +0000 UTC m=+0.630270995 container remove 1d14ff0e495ea18aeb3efae630ad2af8b255b51b51b50169ab3e0077fac846d2 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-17T00:23:11.717 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:23:11 smithi099 systemd[1]: ceph-3aad27a2-7489-11ef-bceb-c7b262605968@prometheus.a.service: Deactivated successfully. 2024-09-17T00:23:11.717 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:23:11 smithi099 systemd[1]: Stopped Ceph prometheus.a for 3aad27a2-7489-11ef-bceb-c7b262605968. 2024-09-17T00:23:11.717 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:23:11 smithi099 systemd[1]: ceph-3aad27a2-7489-11ef-bceb-c7b262605968@prometheus.a.service: Consumed 1.156s CPU time. 2024-09-17T00:23:11.717 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:23:11 smithi099 systemd[1]: Starting Ceph prometheus.a for 3aad27a2-7489-11ef-bceb-c7b262605968... 2024-09-17T00:23:12.000 INFO:teuthology.orchestra.run.smithi002.stdout:HEALTH_OK 2024-09-17T00:23:12.057 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:23:11 smithi099 podman[52742]: 2024-09-17 00:23:11.740700204 +0000 UTC m=+0.017246882 image pull a07b618ecd1dce142bce4c52f0e80982eaf1f14265a2415c2d35978ccaa0a464 quay.io/prometheus/prometheus:v2.43.0 2024-09-17T00:23:12.057 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:23:11 smithi099 podman[52742]: 2024-09-17 00:23:11.864288217 +0000 UTC m=+0.140834895 container create bf32c2ae11dc1ea435bf07e899fff851eaeb008d222407034d1335f0c7d22a0b (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-17T00:23:12.057 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:23:12 smithi099 podman[52742]: 2024-09-17 00:23:12.0331192 +0000 UTC m=+0.309665887 container init bf32c2ae11dc1ea435bf07e899fff851eaeb008d222407034d1335f0c7d22a0b (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-17T00:23:12.057 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:23:12 smithi099 podman[52742]: 2024-09-17 00:23:12.03586756 +0000 UTC m=+0.312414251 container start bf32c2ae11dc1ea435bf07e899fff851eaeb008d222407034d1335f0c7d22a0b (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-17T00:23:12.057 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:23:12 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[52757]: ts=2024-09-17T00:23:12.059Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-09-17T00:23:12.058 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:23:12 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[52757]: ts=2024-09-17T00:23:12.059Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-09-17T00:23:12.088 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:23:12 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[52757]: ts=2024-09-17T00:23:12.059Z caller=main.go:570 level=info host_details="(Linux 5.14.0-508.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024 x86_64 smithi099 (none))" 2024-09-17T00:23:12.088 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:23:12 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[52757]: ts=2024-09-17T00:23:12.059Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-09-17T00:23:12.088 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:23:12 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[52757]: ts=2024-09-17T00:23:12.059Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-09-17T00:23:12.088 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:23:12 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[52757]: ts=2024-09-17T00:23:12.060Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-09-17T00:23:12.088 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:23:12 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[52757]: ts=2024-09-17T00:23:12.061Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-09-17T00:23:12.089 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:23:12 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[52757]: ts=2024-09-17T00:23:12.062Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-09-17T00:23:12.089 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:23:12 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[52757]: ts=2024-09-17T00:23:12.062Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-09-17T00:23:12.089 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:23:12 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[52757]: ts=2024-09-17T00:23:12.064Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-09-17T00:23:12.089 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:23:12 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[52757]: ts=2024-09-17T00:23:12.064Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.68µs 2024-09-17T00:23:12.089 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:23:12 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[52757]: ts=2024-09-17T00:23:12.064Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-09-17T00:23:12.089 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:23:12 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[52757]: ts=2024-09-17T00:23:12.072Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=3 2024-09-17T00:23:12.089 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:23:12 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[52757]: ts=2024-09-17T00:23:12.084Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=3 2024-09-17T00:23:12.089 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:23:12 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[52757]: ts=2024-09-17T00:23:12.091Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=2 maxSegment=3 2024-09-17T00:23:12.089 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:23:12 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[52757]: ts=2024-09-17T00:23:12.092Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=3 maxSegment=3 2024-09-17T00:23:12.089 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:23:12 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[52757]: ts=2024-09-17T00:23:12.092Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=19.366µs wal_replay_duration=27.58252ms wbl_replay_duration=98ns total_replay_duration=27.619589ms 2024-09-17T00:23:12.447 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:23:12 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[52757]: ts=2024-09-17T00:23:12.095Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-09-17T00:23:12.447 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:23:12 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[52757]: ts=2024-09-17T00:23:12.095Z caller=main.go:1029 level=info msg="TSDB started" 2024-09-17T00:23:12.447 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:23:12 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[52757]: ts=2024-09-17T00:23:12.095Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-09-17T00:23:12.447 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:23:12 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[52757]: ts=2024-09-17T00:23:12.114Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=19.133825ms db_storage=839ns remote_storage=1.381µs web_handler=460ns query_engine=644ns scrape=376.392µs scrape_sd=96.353µs notify=16.99µs notify_sd=27.696µs rules=18.309744ms tracing=5.286µs 2024-09-17T00:23:12.447 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:23:12 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[52757]: ts=2024-09-17T00:23:12.114Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-09-17T00:23:12.447 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:23:12 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[52757]: ts=2024-09-17T00:23:12.114Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-09-17T00:23:12.447 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:23:12 smithi099 bash[52742]: bf32c2ae11dc1ea435bf07e899fff851eaeb008d222407034d1335f0c7d22a0b 2024-09-17T00:23:12.447 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:23:12 smithi099 systemd[1]: Started Ceph prometheus.a for 3aad27a2-7489-11ef-bceb-c7b262605968. 2024-09-17T00:23:12.712 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:12 smithi099 ceph-mon[24594]: from='client.25291 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:23:12.712 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:12 smithi099 ceph-mon[24594]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-17T00:23:12.712 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:12 smithi099 ceph-mon[24594]: from='client.? 172.21.15.2:0/2794038916' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T00:23:12.712 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:12 smithi099 ceph-mon[24594]: from='mgr.15333 172.21.15.2:0/842744905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T00:23:12.712 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:12 smithi099 ceph-mon[24594]: from='mgr.15333 ' entity='mgr.y' 2024-09-17T00:23:12.713 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:12 smithi099 ceph-mon[24594]: from='mgr.15333 ' entity='mgr.y' 2024-09-17T00:23:12.713 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:12 smithi099 ceph-mon[24594]: from='mgr.15333 172.21.15.2:0/842744905' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-17T00:23:12.713 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:12 smithi099 ceph-mon[24594]: from='mgr.15333 172.21.15.2:0/842744905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T00:23:12.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:12 smithi002 ceph-mon[23737]: from='client.25291 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:23:12.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:12 smithi002 ceph-mon[23737]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-17T00:23:12.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:12 smithi002 ceph-mon[23737]: from='client.? 172.21.15.2:0/2794038916' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T00:23:12.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:12 smithi002 ceph-mon[23737]: from='mgr.15333 172.21.15.2:0/842744905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T00:23:12.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:12 smithi002 ceph-mon[23737]: from='mgr.15333 ' entity='mgr.y' 2024-09-17T00:23:12.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:12 smithi002 ceph-mon[23737]: from='mgr.15333 ' entity='mgr.y' 2024-09-17T00:23:12.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:12 smithi002 ceph-mon[23737]: from='mgr.15333 172.21.15.2:0/842744905' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-17T00:23:12.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:12 smithi002 ceph-mon[23737]: from='mgr.15333 172.21.15.2:0/842744905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T00:23:12.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:12 smithi002 ceph-mon[28203]: from='client.25291 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:23:12.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:12 smithi002 ceph-mon[28203]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-17T00:23:12.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:12 smithi002 ceph-mon[28203]: from='client.? 172.21.15.2:0/2794038916' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T00:23:12.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:12 smithi002 ceph-mon[28203]: from='mgr.15333 172.21.15.2:0/842744905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T00:23:12.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:12 smithi002 ceph-mon[28203]: from='mgr.15333 ' entity='mgr.y' 2024-09-17T00:23:12.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:12 smithi002 ceph-mon[28203]: from='mgr.15333 ' entity='mgr.y' 2024-09-17T00:23:12.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:12 smithi002 ceph-mon[28203]: from='mgr.15333 172.21.15.2:0/842744905' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-17T00:23:12.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:12 smithi002 ceph-mon[28203]: from='mgr.15333 172.21.15.2:0/842744905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T00:23:13.553 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:13 smithi099 ceph-mon[24594]: from='client.15417 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:23:13.554 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:13 smithi099 ceph-mon[24594]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-17T00:23:13.554 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:23:13 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[52757]: ts=2024-09-17T00:23:13.505Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes name=CephNodeDiskspaceWarning index=4 msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi002\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi002\", job=\"node\", machine=\"x86_64\", nodename=\"smithi002\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"3aad27a2-7489-11ef-bceb-c7b262605968\", domainname=\"(none)\", instance=\"smithi002\", job=\"node\", machine=\"x86_64\", nodename=\"smithi002\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-17T00:23:13.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:13 smithi002 ceph-mon[23737]: from='client.15417 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:23:13.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:13 smithi002 ceph-mon[23737]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-17T00:23:13.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:13 smithi002 ceph-mon[28203]: from='client.15417 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:23:13.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:13 smithi002 ceph-mon[28203]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-17T00:23:14.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:14 smithi099 ceph-mon[24594]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-17T00:23:14.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:14 smithi099 ceph-mon[24594]: from='mgr.15333 172.21.15.2:0/842744905' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-17T00:23:14.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:14 smithi099 ceph-mon[24594]: from='mgr.15333 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-17T00:23:14.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:14 smithi099 ceph-mon[24594]: from='mgr.15333 172.21.15.2:0/842744905' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-17T00:23:14.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:14 smithi099 ceph-mon[24594]: from='mgr.15333 172.21.15.2:0/842744905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:23:14.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:14 smithi002 ceph-mon[28203]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-17T00:23:14.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:14 smithi002 ceph-mon[28203]: from='mgr.15333 172.21.15.2:0/842744905' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-17T00:23:14.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:14 smithi002 ceph-mon[28203]: from='mgr.15333 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-17T00:23:14.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:14 smithi002 ceph-mon[28203]: from='mgr.15333 172.21.15.2:0/842744905' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-17T00:23:14.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:14 smithi002 ceph-mon[28203]: from='mgr.15333 172.21.15.2:0/842744905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:23:14.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:14 smithi002 ceph-mon[23737]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-17T00:23:14.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:14 smithi002 ceph-mon[23737]: from='mgr.15333 172.21.15.2:0/842744905' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-17T00:23:14.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:14 smithi002 ceph-mon[23737]: from='mgr.15333 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-17T00:23:14.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:14 smithi002 ceph-mon[23737]: from='mgr.15333 172.21.15.2:0/842744905' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-17T00:23:14.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:14 smithi002 ceph-mon[23737]: from='mgr.15333 172.21.15.2:0/842744905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:23:15.638 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:15 smithi099 ceph-mon[24594]: Upgrade: Updating mgr.x 2024-09-17T00:23:15.638 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:15 smithi099 ceph-mon[24594]: Deploying daemon mgr.x on smithi099 2024-09-17T00:23:15.638 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:23:15 smithi099 systemd[1]: Stopping Ceph mgr.x for 3aad27a2-7489-11ef-bceb-c7b262605968... 2024-09-17T00:23:15.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:15 smithi002 ceph-mon[28203]: Upgrade: Updating mgr.x 2024-09-17T00:23:15.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:15 smithi002 ceph-mon[28203]: Deploying daemon mgr.x on smithi099 2024-09-17T00:23:15.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:15 smithi002 ceph-mon[23737]: Upgrade: Updating mgr.x 2024-09-17T00:23:15.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:15 smithi002 ceph-mon[23737]: Deploying daemon mgr.x on smithi099 2024-09-17T00:23:15.967 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:23:15 smithi099 podman[53086]: 2024-09-17 00:23:15.640666833 +0000 UTC m=+0.154834836 container died a8e1e8964c84767502f0bfe594cd678554067c241ac97ddd9f857dc76d3f549e (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=squid-c2ddf97, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-09-17T00:23:16.266 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:23:15 smithi099 podman[53086]: 2024-09-17 00:23:15.993168906 +0000 UTC m=+0.507336896 container cleanup a8e1e8964c84767502f0bfe594cd678554067c241ac97ddd9f857dc76d3f549e (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-c2ddf97, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731) 2024-09-17T00:23:16.266 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:23:15 smithi099 bash[53086]: ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x 2024-09-17T00:23:16.606 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:16 smithi099 ceph-mon[24594]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-17T00:23:16.606 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:23:16 smithi099 podman[53100]: 2024-09-17 00:23:16.269574701 +0000 UTC m=+0.626671797 container remove a8e1e8964c84767502f0bfe594cd678554067c241ac97ddd9f857dc76d3f549e (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-c2ddf97, io.buildah.version=1.37.2, ceph=True, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-09-17T00:23:16.606 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:23:16 smithi099 systemd[1]: ceph-3aad27a2-7489-11ef-bceb-c7b262605968@mgr.x.service: Main process exited, code=exited, status=143/n/a 2024-09-17T00:23:16.606 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:23:16 smithi099 systemd[1]: ceph-3aad27a2-7489-11ef-bceb-c7b262605968@mgr.x.service: Failed with result 'exit-code'. 2024-09-17T00:23:16.607 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:23:16 smithi099 systemd[1]: Stopped Ceph mgr.x for 3aad27a2-7489-11ef-bceb-c7b262605968. 2024-09-17T00:23:16.607 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:23:16 smithi099 systemd[1]: ceph-3aad27a2-7489-11ef-bceb-c7b262605968@mgr.x.service: Consumed 18.692s CPU time. 2024-09-17T00:23:16.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:16 smithi002 ceph-mon[28203]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-17T00:23:16.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:16 smithi002 ceph-mon[23737]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-17T00:23:16.951 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:23:16 smithi099 systemd[1]: Starting Ceph mgr.x for 3aad27a2-7489-11ef-bceb-c7b262605968... 2024-09-17T00:23:16.951 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:23:16 smithi099 podman[53226]: 2024-09-17 00:23:16.720843812 +0000 UTC m=+0.016033644 image pull ebb0615149def4a3179c82c0cc47af75808300abb9f1142788e5e605a7ac0e5b quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f 2024-09-17T00:23:16.951 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:23:16 smithi099 podman[53226]: 2024-09-17 00:23:16.836971332 +0000 UTC m=+0.132161161 container create 6aa2ccef9ca06f456bc1be7be15b45be2b21b3f66a345e29eb5261d715dd9fc1 (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x, GIT_CLEAN=True, RELEASE=squid-c2ddf97, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-17T00:23:17.410 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:23:17 smithi099 podman[53226]: 2024-09-17 00:23:17.05601772 +0000 UTC m=+0.351207552 container init 6aa2ccef9ca06f456bc1be7be15b45be2b21b3f66a345e29eb5261d715dd9fc1 (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=squid-c2ddf97, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-17T00:23:17.410 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:23:17 smithi099 podman[53226]: 2024-09-17 00:23:17.058753471 +0000 UTC m=+0.353943294 container start 6aa2ccef9ca06f456bc1be7be15b45be2b21b3f66a345e29eb5261d715dd9fc1 (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, RELEASE=squid-c2ddf97, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-17T00:23:17.410 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:23:17 smithi099 ceph-mgr[53244]: -- 172.21.15.99:0/1094582210 <== mon.2 v2:172.21.15.99:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55e6844f30e0 con 0x55e6844d1000 2024-09-17T00:23:17.410 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:23:17 smithi099 bash[53226]: 6aa2ccef9ca06f456bc1be7be15b45be2b21b3f66a345e29eb5261d715dd9fc1 2024-09-17T00:23:17.411 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:23:17 smithi099 systemd[1]: Started Ceph mgr.x for 3aad27a2-7489-11ef-bceb-c7b262605968. 2024-09-17T00:23:17.411 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:23:17 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[53240]: 2024-09-17T00:23:17.192+0000 7f0b4c05c040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-17T00:23:17.411 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:23:17 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[53240]: 2024-09-17T00:23:17.256+0000 7f0b4c05c040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-17T00:23:17.411 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:23:17 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[53240]: 2024-09-17T00:23:17.411+0000 7f0b4c05c040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-17T00:23:17.717 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:23:17 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[53240]: 2024-09-17T00:23:17.484+0000 7f0b4c05c040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-17T00:23:17.717 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:23:17 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[53240]: 2024-09-17T00:23:17.655+0000 7f0b4c05c040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-17T00:23:18.127 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:23:17 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[53240]: 2024-09-17T00:23:17.801+0000 7f0b4c05c040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-17T00:23:18.390 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:18 smithi099 ceph-mon[24594]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-09-17T00:23:18.391 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:18 smithi099 ceph-mon[24594]: from='mgr.15333 ' entity='mgr.y' 2024-09-17T00:23:18.391 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:18 smithi099 ceph-mon[24594]: from='mgr.15333 ' entity='mgr.y' 2024-09-17T00:23:18.391 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:23:18 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[53240]: 2024-09-17T00:23:18.129+0000 7f0b4c05c040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-17T00:23:18.391 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:23:18 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[53240]: 2024-09-17T00:23:18.189+0000 7f0b4c05c040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-17T00:23:18.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:18 smithi002 ceph-mon[28203]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-09-17T00:23:18.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:18 smithi002 ceph-mon[28203]: from='mgr.15333 ' entity='mgr.y' 2024-09-17T00:23:18.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:18 smithi002 ceph-mon[28203]: from='mgr.15333 ' entity='mgr.y' 2024-09-17T00:23:18.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:18 smithi002 ceph-mon[23737]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-09-17T00:23:18.480 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:18 smithi002 ceph-mon[23737]: from='mgr.15333 ' entity='mgr.y' 2024-09-17T00:23:18.480 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:18 smithi002 ceph-mon[23737]: from='mgr.15333 ' entity='mgr.y' 2024-09-17T00:23:18.665 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:23:18 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[53240]: 2024-09-17T00:23:18.392+0000 7f0b4c05c040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-17T00:23:19.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:19 smithi002 ceph-mon[23737]: from='client.? 172.21.15.2:0/2842686463' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-17T00:23:19.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:19 smithi002 ceph-mon[28203]: from='client.? 172.21.15.2:0/2842686463' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-17T00:23:19.608 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:19 smithi099 ceph-mon[24594]: from='client.? 172.21.15.2:0/2842686463' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-17T00:23:19.608 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:23:19 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[53240]: 2024-09-17T00:23:19.356+0000 7f0b4c05c040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-17T00:23:19.608 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:23:19 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[53240]: 2024-09-17T00:23:19.424+0000 7f0b4c05c040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-17T00:23:20.456 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:20 smithi099 ceph-mon[24594]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-09-17T00:23:20.456 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:23:20 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[53240]: 2024-09-17T00:23:20.183+0000 7f0b4c05c040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-17T00:23:20.456 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:23:20 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[53240]: 2024-09-17T00:23:20.310+0000 7f0b4c05c040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-17T00:23:20.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:20 smithi002 ceph-mon[23737]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-09-17T00:23:20.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:20 smithi002 ceph-mon[28203]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-09-17T00:23:20.717 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:23:20 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[53240]: 2024-09-17T00:23:20.457+0000 7f0b4c05c040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-17T00:23:20.717 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:23:20 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[53240]: 2024-09-17T00:23:20.520+0000 7f0b4c05c040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-17T00:23:20.717 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:23:20 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[53240]: 2024-09-17T00:23:20.681+0000 7f0b4c05c040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-17T00:23:21.044 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:23:20 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[53240]: 2024-09-17T00:23:20.754+0000 7f0b4c05c040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-17T00:23:21.044 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:23:20 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[53240]: 2024-09-17T00:23:20.815+0000 7f0b4c05c040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-17T00:23:21.045 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:23:20 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[53240]: 2024-09-17T00:23:20.980+0000 7f0b4c05c040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-17T00:23:21.045 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:23:21 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[53240]: 2024-09-17T00:23:21.047+0000 7f0b4c05c040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-17T00:23:21.319 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:23:21 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[53240]: 2024-09-17T00:23:21.116+0000 7f0b4c05c040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-17T00:23:21.320 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:23:21 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[53240]: 2024-09-17T00:23:21.201+0000 7f0b4c05c040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-17T00:23:21.320 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:23:21 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[53240]: 2024-09-17T00:23:21.262+0000 7f0b4c05c040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-17T00:23:21.320 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:23:21 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[53240]: 2024-09-17T00:23:21.322+0000 7f0b4c05c040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-17T00:23:22.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:22 smithi099 ceph-mon[24594]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:23:22.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:22 smithi099 ceph-mon[24594]: from='mgr.15333 ' entity='mgr.y' 2024-09-17T00:23:22.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:22 smithi099 ceph-mon[24594]: from='mgr.15333 ' entity='mgr.y' 2024-09-17T00:23:22.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:22 smithi099 ceph-mon[24594]: from='mgr.? 172.21.15.99:0/2553756534' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-17T00:23:22.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:22 smithi099 ceph-mon[24594]: from='mgr.15333 ' entity='mgr.y' 2024-09-17T00:23:22.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:22 smithi099 ceph-mon[24594]: from='mgr.? 172.21.15.99:0/2553756534' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-17T00:23:22.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:22 smithi099 ceph-mon[24594]: Standby manager daemon x restarted 2024-09-17T00:23:22.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:22 smithi099 ceph-mon[24594]: Standby manager daemon x started 2024-09-17T00:23:22.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:22 smithi099 ceph-mon[24594]: from='mgr.? 172.21.15.99:0/2553756534' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-17T00:23:22.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:22 smithi099 ceph-mon[24594]: from='mgr.? 172.21.15.99:0/2553756534' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-17T00:23:22.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:22 smithi099 ceph-mon[24594]: from='mgr.15333 ' entity='mgr.y' 2024-09-17T00:23:22.468 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:23:22 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[53240]: 2024-09-17T00:23:22.179+0000 7f0b4c05c040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-17T00:23:22.468 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:23:22 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[53240]: [17/Sep/2024:00:23:22] ENGINE Bus STARTING 2024-09-17T00:23:22.468 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:23:22 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[53240]: CherryPy Checker: 2024-09-17T00:23:22.468 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:23:22 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[53240]: The Application mounted at '' has an empty config. 2024-09-17T00:23:22.468 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:23:22 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[53240]: 2024-09-17T00:23:22.468 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:23:22 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[53240]: [17/Sep/2024:00:23:22] ENGINE Serving on http://:::9283 2024-09-17T00:23:22.468 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:23:22 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[53240]: [17/Sep/2024:00:23:22] ENGINE Bus STARTED 2024-09-17T00:23:22.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:22 smithi002 ceph-mon[23737]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:23:22.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:22 smithi002 ceph-mon[23737]: from='mgr.15333 ' entity='mgr.y' 2024-09-17T00:23:22.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:22 smithi002 ceph-mon[23737]: from='mgr.15333 ' entity='mgr.y' 2024-09-17T00:23:22.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:22 smithi002 ceph-mon[23737]: from='mgr.? 172.21.15.99:0/2553756534' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-17T00:23:22.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:22 smithi002 ceph-mon[23737]: from='mgr.15333 ' entity='mgr.y' 2024-09-17T00:23:22.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:22 smithi002 ceph-mon[23737]: from='mgr.? 172.21.15.99:0/2553756534' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-17T00:23:22.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:22 smithi002 ceph-mon[23737]: Standby manager daemon x restarted 2024-09-17T00:23:22.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:22 smithi002 ceph-mon[23737]: Standby manager daemon x started 2024-09-17T00:23:22.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:22 smithi002 ceph-mon[23737]: from='mgr.? 172.21.15.99:0/2553756534' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-17T00:23:22.480 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:22 smithi002 ceph-mon[23737]: from='mgr.? 172.21.15.99:0/2553756534' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-17T00:23:22.480 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:22 smithi002 ceph-mon[23737]: from='mgr.15333 ' entity='mgr.y' 2024-09-17T00:23:22.480 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:22 smithi002 ceph-mon[28203]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:23:22.480 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:22 smithi002 ceph-mon[28203]: from='mgr.15333 ' entity='mgr.y' 2024-09-17T00:23:22.480 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:22 smithi002 ceph-mon[28203]: from='mgr.15333 ' entity='mgr.y' 2024-09-17T00:23:22.480 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:22 smithi002 ceph-mon[28203]: from='mgr.? 172.21.15.99:0/2553756534' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-17T00:23:22.480 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:22 smithi002 ceph-mon[28203]: from='mgr.15333 ' entity='mgr.y' 2024-09-17T00:23:22.480 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:22 smithi002 ceph-mon[28203]: from='mgr.? 172.21.15.99:0/2553756534' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-17T00:23:22.480 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:22 smithi002 ceph-mon[28203]: Standby manager daemon x restarted 2024-09-17T00:23:22.480 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:22 smithi002 ceph-mon[28203]: Standby manager daemon x started 2024-09-17T00:23:22.481 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:22 smithi002 ceph-mon[28203]: from='mgr.? 172.21.15.99:0/2553756534' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-17T00:23:22.481 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:22 smithi002 ceph-mon[28203]: from='mgr.? 172.21.15.99:0/2553756534' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-17T00:23:22.481 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:22 smithi002 ceph-mon[28203]: from='mgr.15333 ' entity='mgr.y' 2024-09-17T00:23:23.773 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:23:23 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[52757]: ts=2024-09-17T00:23:23.506Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes name=CephNodeDiskspaceWarning index=4 msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi002\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi002\", job=\"node\", machine=\"x86_64\", nodename=\"smithi002\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"3aad27a2-7489-11ef-bceb-c7b262605968\", domainname=\"(none)\", instance=\"smithi002\", job=\"node\", machine=\"x86_64\", nodename=\"smithi002\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-17T00:23:24.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:24 smithi099 ceph-mon[24594]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:23:24.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:24 smithi099 ceph-mon[24594]: mgrmap e33: y(active, since 27s), standbys: x 2024-09-17T00:23:24.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:24 smithi002 ceph-mon[23737]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:23:24.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:24 smithi002 ceph-mon[23737]: mgrmap e33: y(active, since 27s), standbys: x 2024-09-17T00:23:24.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:24 smithi002 ceph-mon[28203]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:23:24.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:24 smithi002 ceph-mon[28203]: mgrmap e33: y(active, since 27s), standbys: x 2024-09-17T00:23:26.641 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:26 smithi002 ceph-mon[28203]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-09-17T00:23:26.641 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:26 smithi002 ceph-mon[28203]: from='mgr.15333 ' entity='mgr.y' 2024-09-17T00:23:26.641 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:26 smithi002 ceph-mon[28203]: from='mgr.15333 ' entity='mgr.y' 2024-09-17T00:23:26.642 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:26 smithi002 ceph-mon[28203]: from='mgr.15333 172.21.15.2:0/842744905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:23:26.642 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:26 smithi002 ceph-mon[28203]: from='mgr.15333 172.21.15.2:0/842744905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:23:26.642 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:26 smithi002 ceph-mon[28203]: from='mgr.15333 ' entity='mgr.y' 2024-09-17T00:23:26.642 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:26 smithi002 ceph-mon[28203]: from='mgr.15333 172.21.15.2:0/842744905' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi002.vmouem", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-17T00:23:26.642 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:26 smithi002 ceph-mon[28203]: from='mgr.15333 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi002.vmouem", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-17T00:23:26.642 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:26 smithi002 ceph-mon[28203]: from='mgr.15333 172.21.15.2:0/842744905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:23:26.642 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:26 smithi002 ceph-mon[23737]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-09-17T00:23:26.642 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:26 smithi002 ceph-mon[23737]: from='mgr.15333 ' entity='mgr.y' 2024-09-17T00:23:26.642 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:26 smithi002 ceph-mon[23737]: from='mgr.15333 ' entity='mgr.y' 2024-09-17T00:23:26.643 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:26 smithi002 ceph-mon[23737]: from='mgr.15333 172.21.15.2:0/842744905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:23:26.643 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:26 smithi002 ceph-mon[23737]: from='mgr.15333 172.21.15.2:0/842744905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:23:26.643 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:26 smithi002 ceph-mon[23737]: from='mgr.15333 ' entity='mgr.y' 2024-09-17T00:23:26.643 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:26 smithi002 ceph-mon[23737]: from='mgr.15333 172.21.15.2:0/842744905' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi002.vmouem", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-17T00:23:26.643 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:26 smithi002 ceph-mon[23737]: from='mgr.15333 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi002.vmouem", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-17T00:23:26.643 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:26 smithi002 ceph-mon[23737]: from='mgr.15333 172.21.15.2:0/842744905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:23:26.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:26 smithi099 ceph-mon[24594]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-09-17T00:23:26.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:26 smithi099 ceph-mon[24594]: from='mgr.15333 ' entity='mgr.y' 2024-09-17T00:23:26.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:26 smithi099 ceph-mon[24594]: from='mgr.15333 ' entity='mgr.y' 2024-09-17T00:23:26.718 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:26 smithi099 ceph-mon[24594]: from='mgr.15333 172.21.15.2:0/842744905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:23:26.718 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:26 smithi099 ceph-mon[24594]: from='mgr.15333 172.21.15.2:0/842744905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:23:26.718 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:26 smithi099 ceph-mon[24594]: from='mgr.15333 ' entity='mgr.y' 2024-09-17T00:23:26.718 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:26 smithi099 ceph-mon[24594]: from='mgr.15333 172.21.15.2:0/842744905' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi002.vmouem", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-17T00:23:26.718 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:26 smithi099 ceph-mon[24594]: from='mgr.15333 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi002.vmouem", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-17T00:23:26.718 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:26 smithi099 ceph-mon[24594]: from='mgr.15333 172.21.15.2:0/842744905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:23:26.930 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:26 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: ::ffff:172.21.15.99 - - [17/Sep/2024:00:23:26] "GET /metrics HTTP/1.1" 200 38151 "" "Prometheus/2.43.0" 2024-09-17T00:23:27.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:27 smithi002 ceph-mon[28203]: Reconfiguring iscsi.foo.smithi002.vmouem (dependencies changed)... 2024-09-17T00:23:27.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:27 smithi002 ceph-mon[28203]: Reconfiguring daemon iscsi.foo.smithi002.vmouem on smithi002 2024-09-17T00:23:27.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:27 smithi002 ceph-mon[28203]: from='mgr.15333 172.21.15.2:0/842744905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T00:23:27.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:27 smithi002 ceph-mon[23737]: Reconfiguring iscsi.foo.smithi002.vmouem (dependencies changed)... 2024-09-17T00:23:27.480 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:27 smithi002 ceph-mon[23737]: Reconfiguring daemon iscsi.foo.smithi002.vmouem on smithi002 2024-09-17T00:23:27.480 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:27 smithi002 ceph-mon[23737]: from='mgr.15333 172.21.15.2:0/842744905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T00:23:27.704 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:27 smithi099 ceph-mon[24594]: Reconfiguring iscsi.foo.smithi002.vmouem (dependencies changed)... 2024-09-17T00:23:27.704 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:27 smithi099 ceph-mon[24594]: Reconfiguring daemon iscsi.foo.smithi002.vmouem on smithi002 2024-09-17T00:23:27.704 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:27 smithi099 ceph-mon[24594]: from='mgr.15333 172.21.15.2:0/842744905' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T00:23:28.608 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:28 smithi002 ceph-mon[28203]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-09-17T00:23:28.609 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:28 smithi002 ceph-mon[23737]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-09-17T00:23:28.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:28 smithi099 ceph-mon[24594]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-09-17T00:23:28.717 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:23:28 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[52757]: ts=2024-09-17T00:23:28.659Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd name=CephOSDFlapping index=13 msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.2\", device_class=\"ssd\", hostname=\"smithi002\", instance=\"172.21.15.99:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.2\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"3aad27a2-7489-11ef-bceb-c7b262605968\", cluster_addr=\"172.21.15.2\", device_class=\"ssd\", hostname=\"smithi002\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.2\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-17T00:23:28.979 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:28 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: 2024-09-17T00:23:28.689+0000 7f89aca67640 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi002.vmouem ... 2024-09-17T00:23:28.979 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:28 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: Creating ceph-iscsi config... 2024-09-17T00:23:28.980 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:28 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/iscsi-gateway.cfg 2024-09-17T00:23:28.980 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:28 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/tcmu-runner-entrypoint.sh 2024-09-17T00:23:28.980 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:28 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: Non-zero exit code 1 from systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem 2024-09-17T00:23:28.980 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:28 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: systemctl: stderr Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:23:28.980 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:28 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: systemctl: stderr See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:23:28.980 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:28 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: Traceback (most recent call last): 2024-09-17T00:23:28.980 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:28 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-17T00:23:28.980 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:28 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: return _run_code(code, main_globals, None, 2024-09-17T00:23:28.980 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:28 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-17T00:23:28.980 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:28 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: exec(code, run_globals) 2024-09-17T00:23:28.980 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:28 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5579, in 2024-09-17T00:23:28.981 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:28 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5567, in main 2024-09-17T00:23:28.981 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:28 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3049, in command_deploy_from 2024-09-17T00:23:28.981 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:28 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3084, in _common_deploy 2024-09-17T00:23:28.981 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:28 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3104, in _deploy_daemon_container 2024-09-17T00:23:28.981 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:28 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 1124, in deploy_daemon 2024-09-17T00:23:28.981 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:28 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/tmp/tmpnroy8d0w.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-17T00:23:28.981 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:28 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: RuntimeError: Failed command: systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem: Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:23:28.981 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:28 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:23:28.982 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:28 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: Traceback (most recent call last): 2024-09-17T00:23:28.982 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:28 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1134, in _check_daemons 2024-09-17T00:23:28.982 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:28 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: self.mgr._daemon_action(daemon_spec, action=action) 2024-09-17T00:23:28.982 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:28 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2473, in _daemon_action 2024-09-17T00:23:28.982 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:28 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: return self.wait_async( 2024-09-17T00:23:28.982 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:28 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/usr/share/ceph/mgr/cephadm/module.py", line 796, in wait_async 2024-09-17T00:23:28.982 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:28 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: return self.event_loop.get_result(coro, timeout) 2024-09-17T00:23:28.982 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:28 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-09-17T00:23:28.982 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:28 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: return future.result(timeout) 2024-09-17T00:23:28.982 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:28 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/lib64/python3.9/concurrent/futures/_base.py", line 446, in result 2024-09-17T00:23:28.982 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:28 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: return self.__get_result() 2024-09-17T00:23:28.982 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:28 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/lib64/python3.9/concurrent/futures/_base.py", line 391, in __get_result 2024-09-17T00:23:28.983 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:28 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: raise self._exception 2024-09-17T00:23:28.983 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:28 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1370, in _create_daemon 2024-09-17T00:23:28.983 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:28 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: out, err, code = await self._run_cephadm( 2024-09-17T00:23:28.983 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:28 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1713, in _run_cephadm 2024-09-17T00:23:28.983 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:28 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: raise OrchestratorError( 2024-09-17T00:23:28.983 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:28 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi002.vmouem ... 2024-09-17T00:23:28.983 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:28 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: Creating ceph-iscsi config... 2024-09-17T00:23:28.983 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:28 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/iscsi-gateway.cfg 2024-09-17T00:23:28.983 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:28 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/tcmu-runner-entrypoint.sh 2024-09-17T00:23:28.983 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:28 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: Non-zero exit code 1 from systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem 2024-09-17T00:23:28.983 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:28 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: systemctl: stderr Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:23:28.983 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:28 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: systemctl: stderr See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:23:28.983 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:28 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: Traceback (most recent call last): 2024-09-17T00:23:28.984 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:28 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-17T00:23:28.984 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:28 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: return _run_code(code, main_globals, None, 2024-09-17T00:23:28.984 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:28 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-17T00:23:28.984 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:28 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: exec(code, run_globals) 2024-09-17T00:23:28.984 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:28 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5579, in 2024-09-17T00:23:28.984 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:28 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5567, in main 2024-09-17T00:23:28.984 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:28 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3049, in command_deploy_from 2024-09-17T00:23:28.984 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:28 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3084, in _common_deploy 2024-09-17T00:23:28.984 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:28 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3104, in _deploy_daemon_container 2024-09-17T00:23:28.984 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:28 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 1124, in deploy_daemon 2024-09-17T00:23:28.984 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:28 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/tmp/tmpnroy8d0w.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-17T00:23:28.984 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:28 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: RuntimeError: Failed command: systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem: Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:23:28.984 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:28 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:23:29.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:29 smithi099 ceph-mon[24594]: from='mgr.15333 172.21.15.2:0/842744905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T00:23:29.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:29 smithi099 ceph-mon[24594]: from='mgr.15333 172.21.15.2:0/842744905' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:23:29.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:29 smithi099 ceph-mon[24594]: from='mgr.15333 ' entity='mgr.y' 2024-09-17T00:23:29.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:29 smithi099 ceph-mon[24594]: from='mgr.15333 172.21.15.2:0/842744905' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-09-17T00:23:29.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:29 smithi099 ceph-mon[24594]: from='mgr.15333 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-09-17T00:23:29.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:29 smithi099 ceph-mon[24594]: from='mgr.15333 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]': finished 2024-09-17T00:23:29.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:29 smithi099 ceph-mon[24594]: from='mgr.15333 172.21.15.2:0/842744905' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-09-17T00:23:29.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:29 smithi099 ceph-mon[24594]: from='mgr.15333 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-09-17T00:23:29.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:29 smithi099 ceph-mon[24594]: from='mgr.15333 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]': finished 2024-09-17T00:23:29.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:29 smithi099 ceph-mon[24594]: from='mgr.15333 172.21.15.2:0/842744905' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-09-17T00:23:29.718 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:29 smithi099 ceph-mon[24594]: from='mgr.15333 172.21.15.2:0/842744905' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["a"]}]: dispatch 2024-09-17T00:23:29.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:29 smithi002 ceph-mon[28203]: from='mgr.15333 172.21.15.2:0/842744905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T00:23:29.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:29 smithi002 ceph-mon[28203]: from='mgr.15333 172.21.15.2:0/842744905' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:23:29.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:29 smithi002 ceph-mon[28203]: from='mgr.15333 ' entity='mgr.y' 2024-09-17T00:23:29.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:29 smithi002 ceph-mon[28203]: from='mgr.15333 172.21.15.2:0/842744905' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-09-17T00:23:29.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:29 smithi002 ceph-mon[28203]: from='mgr.15333 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-09-17T00:23:29.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:29 smithi002 ceph-mon[28203]: from='mgr.15333 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]': finished 2024-09-17T00:23:29.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:29 smithi002 ceph-mon[28203]: from='mgr.15333 172.21.15.2:0/842744905' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-09-17T00:23:29.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:29 smithi002 ceph-mon[28203]: from='mgr.15333 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-09-17T00:23:29.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:29 smithi002 ceph-mon[28203]: from='mgr.15333 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]': finished 2024-09-17T00:23:29.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:29 smithi002 ceph-mon[28203]: from='mgr.15333 172.21.15.2:0/842744905' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-09-17T00:23:29.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:29 smithi002 ceph-mon[28203]: from='mgr.15333 172.21.15.2:0/842744905' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["a"]}]: dispatch 2024-09-17T00:23:29.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:29 smithi002 ceph-mon[23737]: from='mgr.15333 172.21.15.2:0/842744905' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T00:23:29.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:29 smithi002 ceph-mon[23737]: from='mgr.15333 172.21.15.2:0/842744905' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:23:29.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:29 smithi002 ceph-mon[23737]: from='mgr.15333 ' entity='mgr.y' 2024-09-17T00:23:29.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:29 smithi002 ceph-mon[23737]: from='mgr.15333 172.21.15.2:0/842744905' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-09-17T00:23:29.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:29 smithi002 ceph-mon[23737]: from='mgr.15333 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-09-17T00:23:29.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:29 smithi002 ceph-mon[23737]: from='mgr.15333 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]': finished 2024-09-17T00:23:29.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:29 smithi002 ceph-mon[23737]: from='mgr.15333 172.21.15.2:0/842744905' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-09-17T00:23:29.731 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:29 smithi002 ceph-mon[23737]: from='mgr.15333 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-09-17T00:23:29.731 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:29 smithi002 ceph-mon[23737]: from='mgr.15333 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]': finished 2024-09-17T00:23:29.731 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:29 smithi002 ceph-mon[23737]: from='mgr.15333 172.21.15.2:0/842744905' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-09-17T00:23:29.731 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:29 smithi002 ceph-mon[23737]: from='mgr.15333 172.21.15.2:0/842744905' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["a"]}]: dispatch 2024-09-17T00:23:30.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:30 smithi099 ceph-mon[24594]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi002.vmouem ... 2024-09-17T00:23:30.717 INFO:journalctl@ceph.mon.b.smithi099.stdout: Creating ceph-iscsi config... 2024-09-17T00:23:30.717 INFO:journalctl@ceph.mon.b.smithi099.stdout: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/iscsi-gateway.cfg 2024-09-17T00:23:30.718 INFO:journalctl@ceph.mon.b.smithi099.stdout: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/tcmu-runner-entrypoint.sh 2024-09-17T00:23:30.718 INFO:journalctl@ceph.mon.b.smithi099.stdout: Non-zero exit code 1 from systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem 2024-09-17T00:23:30.718 INFO:journalctl@ceph.mon.b.smithi099.stdout: systemctl: stderr Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:23:30.718 INFO:journalctl@ceph.mon.b.smithi099.stdout: systemctl: stderr See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:23:30.718 INFO:journalctl@ceph.mon.b.smithi099.stdout: Traceback (most recent call last): 2024-09-17T00:23:30.718 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-17T00:23:30.718 INFO:journalctl@ceph.mon.b.smithi099.stdout: return _run_code(code, main_globals, None, 2024-09-17T00:23:30.718 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-17T00:23:30.718 INFO:journalctl@ceph.mon.b.smithi099.stdout: exec(code, run_globals) 2024-09-17T00:23:30.718 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5579, in 2024-09-17T00:23:30.718 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5567, in main 2024-09-17T00:23:30.718 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3049, in command_deploy_from 2024-09-17T00:23:30.719 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3084, in _common_deploy 2024-09-17T00:23:30.719 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3104, in _deploy_daemon_container 2024-09-17T00:23:30.719 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 1124, in deploy_daemon 2024-09-17T00:23:30.719 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-17T00:23:30.719 INFO:journalctl@ceph.mon.b.smithi099.stdout: RuntimeError: Failed command: systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem: Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:23:30.719 INFO:journalctl@ceph.mon.b.smithi099.stdout: See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:23:30.719 INFO:journalctl@ceph.mon.b.smithi099.stdout: Traceback (most recent call last): 2024-09-17T00:23:30.719 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1134, in _check_daemons 2024-09-17T00:23:30.719 INFO:journalctl@ceph.mon.b.smithi099.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-09-17T00:23:30.719 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2473, in _daemon_action 2024-09-17T00:23:30.719 INFO:journalctl@ceph.mon.b.smithi099.stdout: return self.wait_async( 2024-09-17T00:23:30.719 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 796, in wait_async 2024-09-17T00:23:30.719 INFO:journalctl@ceph.mon.b.smithi099.stdout: return self.event_loop.get_result(coro, timeout) 2024-09-17T00:23:30.720 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-09-17T00:23:30.720 INFO:journalctl@ceph.mon.b.smithi099.stdout: return future.result(timeout) 2024-09-17T00:23:30.720 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/lib64/python3.9/concurrent/futures/_base.py", line 446, in result 2024-09-17T00:23:30.720 INFO:journalctl@ceph.mon.b.smithi099.stdout: return self.__get_result() 2024-09-17T00:23:30.720 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/lib64/python3.9/concurrent/futures/_base.py", line 391, in __get_result 2024-09-17T00:23:30.720 INFO:journalctl@ceph.mon.b.smithi099.stdout: raise self._exception 2024-09-17T00:23:30.720 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1370, in _create_daemon 2024-09-17T00:23:30.720 INFO:journalctl@ceph.mon.b.smithi099.stdout: out, err, code = await self._run_cephadm( 2024-09-17T00:23:30.720 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1713, in _run_cephadm 2024-09-17T00:23:30.720 INFO:journalctl@ceph.mon.b.smithi099.stdout: raise OrchestratorError( 2024-09-17T00:23:30.720 INFO:journalctl@ceph.mon.b.smithi099.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi002.vmouem ... 2024-09-17T00:23:30.720 INFO:journalctl@ceph.mon.b.smithi099.stdout: Creating ceph-iscsi config... 2024-09-17T00:23:30.720 INFO:journalctl@ceph.mon.b.smithi099.stdout: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/iscsi-gateway.cfg 2024-09-17T00:23:30.720 INFO:journalctl@ceph.mon.b.smithi099.stdout: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/tcmu-runner-entrypoint.sh 2024-09-17T00:23:30.721 INFO:journalctl@ceph.mon.b.smithi099.stdout: Non-zero exit code 1 from systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem 2024-09-17T00:23:30.721 INFO:journalctl@ceph.mon.b.smithi099.stdout: systemctl: stderr Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:23:30.721 INFO:journalctl@ceph.mon.b.smithi099.stdout: systemctl: stderr See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:23:30.721 INFO:journalctl@ceph.mon.b.smithi099.stdout: Traceback (most recent call last): 2024-09-17T00:23:30.721 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-17T00:23:30.721 INFO:journalctl@ceph.mon.b.smithi099.stdout: return _run_code(code, main_globals, None, 2024-09-17T00:23:30.721 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-17T00:23:30.721 INFO:journalctl@ceph.mon.b.smithi099.stdout: exec(code, run_globals) 2024-09-17T00:23:30.721 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5579, in 2024-09-17T00:23:30.721 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5567, in main 2024-09-17T00:23:30.721 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3049, in command_deploy_from 2024-09-17T00:23:30.721 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3084, in _common_deploy 2024-09-17T00:23:30.722 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3104, in _deploy_daemon_container 2024-09-17T00:23:30.722 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 1124, in deploy_daemon 2024-09-17T00:23:30.722 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-17T00:23:30.722 INFO:journalctl@ceph.mon.b.smithi099.stdout: RuntimeError: Failed command: systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem: Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:23:30.722 INFO:journalctl@ceph.mon.b.smithi099.stdout: See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:23:30.722 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:30 smithi099 ceph-mon[24594]: Upgrade: Setting container_image for all mgr 2024-09-17T00:23:30.722 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:30 smithi099 ceph-mon[24594]: Upgrade: It appears safe to stop mon.a 2024-09-17T00:23:30.722 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:30 smithi099 ceph-mon[24594]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:23:30.722 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:30 smithi099 ceph-mon[24594]: from='mgr.15333 ' entity='mgr.y' 2024-09-17T00:23:30.722 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:30 smithi099 ceph-mon[24594]: from='mgr.15333 172.21.15.2:0/842744905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-17T00:23:30.722 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:30 smithi099 ceph-mon[24594]: from='mgr.15333 172.21.15.2:0/842744905' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-17T00:23:30.722 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:30 smithi099 ceph-mon[24594]: from='mgr.15333 172.21.15.2:0/842744905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:23:30.723 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:30 smithi002 ceph-mon[23737]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi002.vmouem ... 2024-09-17T00:23:30.723 INFO:journalctl@ceph.mon.a.smithi002.stdout: Creating ceph-iscsi config... 2024-09-17T00:23:30.723 INFO:journalctl@ceph.mon.a.smithi002.stdout: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/iscsi-gateway.cfg 2024-09-17T00:23:30.723 INFO:journalctl@ceph.mon.a.smithi002.stdout: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/tcmu-runner-entrypoint.sh 2024-09-17T00:23:30.724 INFO:journalctl@ceph.mon.a.smithi002.stdout: Non-zero exit code 1 from systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem 2024-09-17T00:23:30.724 INFO:journalctl@ceph.mon.a.smithi002.stdout: systemctl: stderr Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:23:30.724 INFO:journalctl@ceph.mon.a.smithi002.stdout: systemctl: stderr See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:23:30.724 INFO:journalctl@ceph.mon.a.smithi002.stdout: Traceback (most recent call last): 2024-09-17T00:23:30.724 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-17T00:23:30.725 INFO:journalctl@ceph.mon.a.smithi002.stdout: return _run_code(code, main_globals, None, 2024-09-17T00:23:30.725 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-17T00:23:30.725 INFO:journalctl@ceph.mon.a.smithi002.stdout: exec(code, run_globals) 2024-09-17T00:23:30.725 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5579, in 2024-09-17T00:23:30.725 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5567, in main 2024-09-17T00:23:30.725 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3049, in command_deploy_from 2024-09-17T00:23:30.725 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3084, in _common_deploy 2024-09-17T00:23:30.725 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3104, in _deploy_daemon_container 2024-09-17T00:23:30.725 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 1124, in deploy_daemon 2024-09-17T00:23:30.725 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-17T00:23:30.725 INFO:journalctl@ceph.mon.a.smithi002.stdout: RuntimeError: Failed command: systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem: Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:23:30.725 INFO:journalctl@ceph.mon.a.smithi002.stdout: See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:23:30.726 INFO:journalctl@ceph.mon.a.smithi002.stdout: Traceback (most recent call last): 2024-09-17T00:23:30.726 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1134, in _check_daemons 2024-09-17T00:23:30.726 INFO:journalctl@ceph.mon.a.smithi002.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-09-17T00:23:30.726 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2473, in _daemon_action 2024-09-17T00:23:30.726 INFO:journalctl@ceph.mon.a.smithi002.stdout: return self.wait_async( 2024-09-17T00:23:30.726 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 796, in wait_async 2024-09-17T00:23:30.726 INFO:journalctl@ceph.mon.a.smithi002.stdout: return self.event_loop.get_result(coro, timeout) 2024-09-17T00:23:30.726 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-09-17T00:23:30.726 INFO:journalctl@ceph.mon.a.smithi002.stdout: return future.result(timeout) 2024-09-17T00:23:30.726 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/lib64/python3.9/concurrent/futures/_base.py", line 446, in result 2024-09-17T00:23:30.726 INFO:journalctl@ceph.mon.a.smithi002.stdout: return self.__get_result() 2024-09-17T00:23:30.726 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/lib64/python3.9/concurrent/futures/_base.py", line 391, in __get_result 2024-09-17T00:23:30.726 INFO:journalctl@ceph.mon.a.smithi002.stdout: raise self._exception 2024-09-17T00:23:30.726 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1370, in _create_daemon 2024-09-17T00:23:30.727 INFO:journalctl@ceph.mon.a.smithi002.stdout: out, err, code = await self._run_cephadm( 2024-09-17T00:23:30.727 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1713, in _run_cephadm 2024-09-17T00:23:30.727 INFO:journalctl@ceph.mon.a.smithi002.stdout: raise OrchestratorError( 2024-09-17T00:23:30.727 INFO:journalctl@ceph.mon.a.smithi002.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi002.vmouem ... 2024-09-17T00:23:30.727 INFO:journalctl@ceph.mon.a.smithi002.stdout: Creating ceph-iscsi config... 2024-09-17T00:23:30.727 INFO:journalctl@ceph.mon.a.smithi002.stdout: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/iscsi-gateway.cfg 2024-09-17T00:23:30.727 INFO:journalctl@ceph.mon.a.smithi002.stdout: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/tcmu-runner-entrypoint.sh 2024-09-17T00:23:30.727 INFO:journalctl@ceph.mon.a.smithi002.stdout: Non-zero exit code 1 from systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem 2024-09-17T00:23:30.727 INFO:journalctl@ceph.mon.a.smithi002.stdout: systemctl: stderr Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:23:30.727 INFO:journalctl@ceph.mon.a.smithi002.stdout: systemctl: stderr See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:23:30.727 INFO:journalctl@ceph.mon.a.smithi002.stdout: Traceback (most recent call last): 2024-09-17T00:23:30.727 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-17T00:23:30.728 INFO:journalctl@ceph.mon.a.smithi002.stdout: return _run_code(code, main_globals, None, 2024-09-17T00:23:30.728 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-17T00:23:30.728 INFO:journalctl@ceph.mon.a.smithi002.stdout: exec(code, run_globals) 2024-09-17T00:23:30.728 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5579, in 2024-09-17T00:23:30.728 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5567, in main 2024-09-17T00:23:30.728 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3049, in command_deploy_from 2024-09-17T00:23:30.728 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3084, in _common_deploy 2024-09-17T00:23:30.728 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3104, in _deploy_daemon_container 2024-09-17T00:23:30.728 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 1124, in deploy_daemon 2024-09-17T00:23:30.728 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-17T00:23:30.729 INFO:journalctl@ceph.mon.a.smithi002.stdout: RuntimeError: Failed command: systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem: Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:23:30.729 INFO:journalctl@ceph.mon.a.smithi002.stdout: See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:23:30.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:30 smithi002 ceph-mon[23737]: Upgrade: Setting container_image for all mgr 2024-09-17T00:23:30.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:30 smithi002 ceph-mon[23737]: Upgrade: It appears safe to stop mon.a 2024-09-17T00:23:30.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:30 smithi002 ceph-mon[28203]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi002.vmouem ... 2024-09-17T00:23:30.729 INFO:journalctl@ceph.mon.c.smithi002.stdout: Creating ceph-iscsi config... 2024-09-17T00:23:30.729 INFO:journalctl@ceph.mon.c.smithi002.stdout: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/iscsi-gateway.cfg 2024-09-17T00:23:30.729 INFO:journalctl@ceph.mon.c.smithi002.stdout: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/tcmu-runner-entrypoint.sh 2024-09-17T00:23:30.729 INFO:journalctl@ceph.mon.c.smithi002.stdout: Non-zero exit code 1 from systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem 2024-09-17T00:23:30.729 INFO:journalctl@ceph.mon.c.smithi002.stdout: systemctl: stderr Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:23:30.730 INFO:journalctl@ceph.mon.c.smithi002.stdout: systemctl: stderr See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:23:30.730 INFO:journalctl@ceph.mon.c.smithi002.stdout: Traceback (most recent call last): 2024-09-17T00:23:30.730 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-17T00:23:30.730 INFO:journalctl@ceph.mon.c.smithi002.stdout: return _run_code(code, main_globals, None, 2024-09-17T00:23:30.730 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-17T00:23:30.730 INFO:journalctl@ceph.mon.c.smithi002.stdout: exec(code, run_globals) 2024-09-17T00:23:30.730 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5579, in 2024-09-17T00:23:30.730 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5567, in main 2024-09-17T00:23:30.730 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3049, in command_deploy_from 2024-09-17T00:23:30.730 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3084, in _common_deploy 2024-09-17T00:23:30.730 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3104, in _deploy_daemon_container 2024-09-17T00:23:30.730 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 1124, in deploy_daemon 2024-09-17T00:23:30.730 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-17T00:23:30.730 INFO:journalctl@ceph.mon.c.smithi002.stdout: RuntimeError: Failed command: systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem: Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:23:30.731 INFO:journalctl@ceph.mon.c.smithi002.stdout: See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:23:30.731 INFO:journalctl@ceph.mon.c.smithi002.stdout: Traceback (most recent call last): 2024-09-17T00:23:30.731 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1134, in _check_daemons 2024-09-17T00:23:30.731 INFO:journalctl@ceph.mon.c.smithi002.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-09-17T00:23:30.731 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2473, in _daemon_action 2024-09-17T00:23:30.731 INFO:journalctl@ceph.mon.c.smithi002.stdout: return self.wait_async( 2024-09-17T00:23:30.731 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 796, in wait_async 2024-09-17T00:23:30.731 INFO:journalctl@ceph.mon.c.smithi002.stdout: return self.event_loop.get_result(coro, timeout) 2024-09-17T00:23:30.731 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-09-17T00:23:30.731 INFO:journalctl@ceph.mon.c.smithi002.stdout: return future.result(timeout) 2024-09-17T00:23:30.731 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/lib64/python3.9/concurrent/futures/_base.py", line 446, in result 2024-09-17T00:23:30.732 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:30 smithi002 ceph-mon[23737]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:23:30.732 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:30 smithi002 ceph-mon[23737]: from='mgr.15333 ' entity='mgr.y' 2024-09-17T00:23:30.732 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:30 smithi002 ceph-mon[23737]: from='mgr.15333 172.21.15.2:0/842744905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-17T00:23:30.732 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:30 smithi002 ceph-mon[23737]: from='mgr.15333 172.21.15.2:0/842744905' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-17T00:23:30.732 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:30 smithi002 ceph-mon[23737]: from='mgr.15333 172.21.15.2:0/842744905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:23:30.733 INFO:journalctl@ceph.mon.c.smithi002.stdout: return self.__get_result() 2024-09-17T00:23:30.733 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/lib64/python3.9/concurrent/futures/_base.py", line 391, in __get_result 2024-09-17T00:23:30.733 INFO:journalctl@ceph.mon.c.smithi002.stdout: raise self._exception 2024-09-17T00:23:30.733 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1370, in _create_daemon 2024-09-17T00:23:30.733 INFO:journalctl@ceph.mon.c.smithi002.stdout: out, err, code = await self._run_cephadm( 2024-09-17T00:23:30.733 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1713, in _run_cephadm 2024-09-17T00:23:30.733 INFO:journalctl@ceph.mon.c.smithi002.stdout: raise OrchestratorError( 2024-09-17T00:23:30.733 INFO:journalctl@ceph.mon.c.smithi002.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi002.vmouem ... 2024-09-17T00:23:30.733 INFO:journalctl@ceph.mon.c.smithi002.stdout: Creating ceph-iscsi config... 2024-09-17T00:23:30.734 INFO:journalctl@ceph.mon.c.smithi002.stdout: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/iscsi-gateway.cfg 2024-09-17T00:23:30.734 INFO:journalctl@ceph.mon.c.smithi002.stdout: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/tcmu-runner-entrypoint.sh 2024-09-17T00:23:30.734 INFO:journalctl@ceph.mon.c.smithi002.stdout: Non-zero exit code 1 from systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem 2024-09-17T00:23:30.734 INFO:journalctl@ceph.mon.c.smithi002.stdout: systemctl: stderr Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:23:30.734 INFO:journalctl@ceph.mon.c.smithi002.stdout: systemctl: stderr See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:23:30.734 INFO:journalctl@ceph.mon.c.smithi002.stdout: Traceback (most recent call last): 2024-09-17T00:23:30.734 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-17T00:23:30.734 INFO:journalctl@ceph.mon.c.smithi002.stdout: return _run_code(code, main_globals, None, 2024-09-17T00:23:30.734 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-17T00:23:30.734 INFO:journalctl@ceph.mon.c.smithi002.stdout: exec(code, run_globals) 2024-09-17T00:23:30.734 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5579, in 2024-09-17T00:23:30.734 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5567, in main 2024-09-17T00:23:30.734 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3049, in command_deploy_from 2024-09-17T00:23:30.734 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3084, in _common_deploy 2024-09-17T00:23:30.735 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3104, in _deploy_daemon_container 2024-09-17T00:23:30.735 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 1124, in deploy_daemon 2024-09-17T00:23:30.735 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-17T00:23:30.735 INFO:journalctl@ceph.mon.c.smithi002.stdout: RuntimeError: Failed command: systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem: Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:23:30.735 INFO:journalctl@ceph.mon.c.smithi002.stdout: See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:23:30.735 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:30 smithi002 ceph-mon[28203]: Upgrade: Setting container_image for all mgr 2024-09-17T00:23:30.735 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:30 smithi002 ceph-mon[28203]: Upgrade: It appears safe to stop mon.a 2024-09-17T00:23:30.735 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:30 smithi002 ceph-mon[28203]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:23:30.735 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:30 smithi002 ceph-mon[28203]: from='mgr.15333 ' entity='mgr.y' 2024-09-17T00:23:30.735 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:30 smithi002 ceph-mon[28203]: from='mgr.15333 172.21.15.2:0/842744905' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-17T00:23:30.735 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:30 smithi002 ceph-mon[28203]: from='mgr.15333 172.21.15.2:0/842744905' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-17T00:23:30.735 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:30 smithi002 ceph-mon[28203]: from='mgr.15333 172.21.15.2:0/842744905' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:23:31.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:31 smithi099 ceph-mon[24594]: Upgrade: Updating mon.a 2024-09-17T00:23:31.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:31 smithi099 ceph-mon[24594]: Deploying daemon mon.a on smithi002 2024-09-17T00:23:31.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:31 smithi002 ceph-mon[28203]: Upgrade: Updating mon.a 2024-09-17T00:23:31.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:31 smithi002 ceph-mon[28203]: Deploying daemon mon.a on smithi002 2024-09-17T00:23:31.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:31 smithi002 ceph-mon[23737]: Upgrade: Updating mon.a 2024-09-17T00:23:31.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:31 smithi002 ceph-mon[23737]: Deploying daemon mon.a on smithi002 2024-09-17T00:23:31.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:31 smithi002 systemd[1]: Stopping Ceph mon.a for 3aad27a2-7489-11ef-bceb-c7b262605968... 2024-09-17T00:23:32.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:31 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mon-a[23733]: 2024-09-17T00:23:31.896+0000 7f215442a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-17T00:23:32.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:31 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mon-a[23733]: 2024-09-17T00:23:31.896+0000 7f215442a700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-09-17T00:23:32.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:31 smithi002 podman[63646]: 2024-09-17 00:23:31.937477102 +0000 UTC m=+0.134163709 container died a68823df625f16cc31a5afdf232d2cb15ad7217d72d4553d92573f32439f1f15 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mon-a, io.k8s.display-name=CentOS Stream 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, ceph=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, distribution-scope=public, io.openshift.expose-services=, io.openshift.tags=base centos centos-stream, build-date=2022-05-03T08:36:31.336870, vendor=Red Hat, Inc., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_REPO=https://github.com/ceph/ceph-container.git, release=754, CEPH_POINT_RELEASE=-17.2.0, io.buildah.version=1.19.8, GIT_CLEAN=True, RELEASE=HEAD, GIT_BRANCH=HEAD, com.redhat.component=centos-stream-container, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-type=git, architecture=x86_64, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, version=8, name=centos-stream, maintainer=Guillaume Abrioux ) 2024-09-17T00:23:32.611 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:32 smithi002 podman[63646]: 2024-09-17 00:23:32.321058867 +0000 UTC m=+0.517745471 container cleanup a68823df625f16cc31a5afdf232d2cb15ad7217d72d4553d92573f32439f1f15 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mon-a, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, build-date=2022-05-03T08:36:31.336870, io.openshift.expose-services=, io.k8s.display-name=CentOS Stream 8, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.component=centos-stream-container, CEPH_POINT_RELEASE=-17.2.0, architecture=x86_64, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.19.8, version=8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, RELEASE=HEAD, ceph=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, io.openshift.tags=base centos centos-stream, name=centos-stream, vcs-type=git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vendor=Red Hat, Inc., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, release=754) 2024-09-17T00:23:32.611 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:32 smithi002 bash[63646]: ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mon-a 2024-09-17T00:23:32.956 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:32 smithi002 podman[63659]: 2024-09-17 00:23:32.614109177 +0000 UTC m=+0.679749390 container remove a68823df625f16cc31a5afdf232d2cb15ad7217d72d4553d92573f32439f1f15 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mon-a, com.redhat.component=centos-stream-container, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.buildah.version=1.19.8, version=8, GIT_CLEAN=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=centos-stream, maintainer=Guillaume Abrioux , com.redhat.license_terms=https://centos.org/legal/licensing-policy/, architecture=x86_64, distribution-scope=public, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_BRANCH=HEAD, vendor=Red Hat, Inc., RELEASE=HEAD, io.openshift.expose-services=, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, ceph=True, CEPH_POINT_RELEASE=-17.2.0, io.k8s.display-name=CentOS Stream 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_REPO=https://github.com/ceph/ceph-container.git, build-date=2022-05-03T08:36:31.336870, io.openshift.tags=base centos centos-stream, release=754, vcs-type=git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image) 2024-09-17T00:23:32.956 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:32 smithi002 systemd[1]: ceph-3aad27a2-7489-11ef-bceb-c7b262605968@mon.a.service: Deactivated successfully. 2024-09-17T00:23:32.956 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:32 smithi002 systemd[1]: Stopped Ceph mon.a for 3aad27a2-7489-11ef-bceb-c7b262605968. 2024-09-17T00:23:32.956 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:32 smithi002 systemd[1]: ceph-3aad27a2-7489-11ef-bceb-c7b262605968@mon.a.service: Consumed 15.829s CPU time. 2024-09-17T00:23:32.956 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:32 smithi002 systemd[1]: Starting Ceph mon.a for 3aad27a2-7489-11ef-bceb-c7b262605968... 2024-09-17T00:23:33.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 podman[63785]: 2024-09-17 00:23:33.072631896 +0000 UTC m=+0.017020760 image pull ebb0615149def4a3179c82c0cc47af75808300abb9f1142788e5e605a7ac0e5b quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f 2024-09-17T00:23:33.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 podman[63785]: 2024-09-17 00:23:33.198612751 +0000 UTC m=+0.143001603 container create a035def53ef68b141bb6821552dcaaad986ad917acefceef91af915e462c773c (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mon-a, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-c2ddf97, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-17T00:23:33.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 podman[63785]: 2024-09-17 00:23:33.426448795 +0000 UTC m=+0.370837661 container init a035def53ef68b141bb6821552dcaaad986ad917acefceef91af915e462c773c (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mon-a, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-c2ddf97, CEPH_POINT_RELEASE=) 2024-09-17T00:23:33.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 podman[63785]: 2024-09-17 00:23:33.430405578 +0000 UTC m=+0.374794430 container start a035def53ef68b141bb6821552dcaaad986ad917acefceef91af915e462c773c (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mon-a, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, RELEASE=squid-c2ddf97, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-17T00:23:33.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: set uid:gid to 167:167 (ceph:ceph) 2024-09-17T00:23:33.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: ceph version 19.1.1-330-gc2ddf972 (c2ddf9721ef254645201cdbabd9c8f49c30a586f) squid (stable), process ceph-mon, pid 2 2024-09-17T00:23:33.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: pidfile_write: ignore empty --pid-file 2024-09-17T00:23:33.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: load: jerasure load: lrc 2024-09-17T00:23:33.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: RocksDB version: 7.9.2 2024-09-17T00:23:33.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Git sha 0 2024-09-17T00:23:33.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Compile date 2024-09-16 15:09:42 2024-09-17T00:23:33.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: DB SUMMARY 2024-09-17T00:23:33.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: DB Session ID: PE7BPYGZLXW1ES7SVVVL 2024-09-17T00:23:33.731 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: CURRENT file: CURRENT 2024-09-17T00:23:33.731 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: IDENTITY file: IDENTITY 2024-09-17T00:23:33.731 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: MANIFEST file: MANIFEST-000015 size: 972 Bytes 2024-09-17T00:23:33.731 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000030.sst 2024-09-17T00:23:33.731 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000028.log size: 3530007 ; 2024-09-17T00:23:33.731 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.error_if_exists: 0 2024-09-17T00:23:33.731 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.create_if_missing: 0 2024-09-17T00:23:33.731 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.paranoid_checks: 1 2024-09-17T00:23:33.731 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.flush_verify_memtable_count: 1 2024-09-17T00:23:33.732 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-17T00:23:33.732 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-09-17T00:23:33.732 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.env: 0x55bc589eb940 2024-09-17T00:23:33.732 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.fs: PosixFileSystem 2024-09-17T00:23:33.732 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.info_log: 0x55bc5956d6e0 2024-09-17T00:23:33.732 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.max_file_opening_threads: 16 2024-09-17T00:23:33.732 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.statistics: (nil) 2024-09-17T00:23:33.734 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.use_fsync: 0 2024-09-17T00:23:33.734 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.max_log_file_size: 0 2024-09-17T00:23:33.734 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-17T00:23:33.734 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-17T00:23:33.734 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.keep_log_file_num: 1000 2024-09-17T00:23:33.734 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.recycle_log_file_num: 0 2024-09-17T00:23:33.734 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.allow_fallocate: 1 2024-09-17T00:23:33.734 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.allow_mmap_reads: 0 2024-09-17T00:23:33.735 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.allow_mmap_writes: 0 2024-09-17T00:23:33.735 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.use_direct_reads: 0 2024-09-17T00:23:33.735 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-17T00:23:33.735 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.create_missing_column_families: 0 2024-09-17T00:23:33.735 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.db_log_dir: 2024-09-17T00:23:33.735 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.wal_dir: 2024-09-17T00:23:33.735 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-17T00:23:33.735 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-17T00:23:33.735 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-17T00:23:33.735 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-17T00:23:33.735 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-17T00:23:33.735 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-17T00:23:33.735 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.advise_random_on_open: 1 2024-09-17T00:23:33.736 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.db_write_buffer_size: 0 2024-09-17T00:23:33.736 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.write_buffer_manager: 0x55bc5957cdc0 2024-09-17T00:23:33.736 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-17T00:23:33.736 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-17T00:23:33.736 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-17T00:23:33.736 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.rate_limiter: (nil) 2024-09-17T00:23:33.736 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-17T00:23:33.736 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.wal_recovery_mode: 2 2024-09-17T00:23:33.736 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.enable_thread_tracking: 0 2024-09-17T00:23:33.736 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.enable_pipelined_write: 0 2024-09-17T00:23:33.736 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.unordered_write: 0 2024-09-17T00:23:33.736 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-17T00:23:33.736 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-17T00:23:33.736 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-17T00:23:33.736 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-17T00:23:33.736 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.row_cache: None 2024-09-17T00:23:33.737 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.wal_filter: None 2024-09-17T00:23:33.737 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-17T00:23:33.737 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.allow_ingest_behind: 0 2024-09-17T00:23:33.737 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.two_write_queues: 0 2024-09-17T00:23:33.737 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.manual_wal_flush: 0 2024-09-17T00:23:33.737 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.wal_compression: 0 2024-09-17T00:23:33.737 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.atomic_flush: 0 2024-09-17T00:23:33.737 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-17T00:23:33.737 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-17T00:23:33.737 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-17T00:23:33.737 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.log_readahead_size: 0 2024-09-17T00:23:33.737 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-17T00:23:33.737 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.best_efforts_recovery: 0 2024-09-17T00:23:33.738 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-17T00:23:33.738 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-17T00:23:33.738 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.allow_data_in_errors: 0 2024-09-17T00:23:33.738 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.db_host_id: __hostname__ 2024-09-17T00:23:33.738 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.enforce_single_del_contracts: true 2024-09-17T00:23:33.738 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.max_background_jobs: 2 2024-09-17T00:23:33.738 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.max_background_compactions: -1 2024-09-17T00:23:33.738 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.max_subcompactions: 1 2024-09-17T00:23:33.738 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-17T00:23:33.738 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-17T00:23:33.738 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-17T00:23:33.738 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.max_total_wal_size: 0 2024-09-17T00:23:33.738 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-17T00:23:33.738 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-17T00:23:33.738 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-17T00:23:33.738 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-17T00:23:33.739 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.max_open_files: -1 2024-09-17T00:23:33.739 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.bytes_per_sync: 0 2024-09-17T00:23:33.739 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-17T00:23:33.739 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-17T00:23:33.739 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.compaction_readahead_size: 0 2024-09-17T00:23:33.739 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.max_background_flushes: -1 2024-09-17T00:23:33.739 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Compression algorithms supported: 2024-09-17T00:23:33.739 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: kZSTD supported: 0 2024-09-17T00:23:33.739 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: kXpressCompression supported: 0 2024-09-17T00:23:33.739 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: kBZip2Compression supported: 0 2024-09-17T00:23:33.739 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-17T00:23:33.739 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: kLZ4Compression supported: 1 2024-09-17T00:23:33.739 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: kZlibCompression supported: 1 2024-09-17T00:23:33.739 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: kLZ4HCCompression supported: 1 2024-09-17T00:23:33.739 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: kSnappyCompression supported: 1 2024-09-17T00:23:33.740 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-17T00:23:33.740 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: DMutex implementation: pthread_mutex_t 2024-09-17T00:23:33.740 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000015 2024-09-17T00:23:33.740 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-09-17T00:23:33.740 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-17T00:23:33.740 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.merge_operator: 2024-09-17T00:23:33.740 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.compaction_filter: None 2024-09-17T00:23:33.740 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.compaction_filter_factory: None 2024-09-17T00:23:33.740 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.sst_partitioner_factory: None 2024-09-17T00:23:33.740 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-17T00:23:33.740 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-17T00:23:33.740 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55bc5956d320) 2024-09-17T00:23:33.740 INFO:journalctl@ceph.mon.a.smithi002.stdout: cache_index_and_filter_blocks: 1 2024-09-17T00:23:33.740 INFO:journalctl@ceph.mon.a.smithi002.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-17T00:23:33.741 INFO:journalctl@ceph.mon.a.smithi002.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-17T00:23:33.741 INFO:journalctl@ceph.mon.a.smithi002.stdout: pin_top_level_index_and_filter: 1 2024-09-17T00:23:33.741 INFO:journalctl@ceph.mon.a.smithi002.stdout: index_type: 0 2024-09-17T00:23:33.741 INFO:journalctl@ceph.mon.a.smithi002.stdout: data_block_index_type: 0 2024-09-17T00:23:33.741 INFO:journalctl@ceph.mon.a.smithi002.stdout: index_shortening: 1 2024-09-17T00:23:33.741 INFO:journalctl@ceph.mon.a.smithi002.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-17T00:23:33.741 INFO:journalctl@ceph.mon.a.smithi002.stdout: checksum: 4 2024-09-17T00:23:33.741 INFO:journalctl@ceph.mon.a.smithi002.stdout: no_block_cache: 0 2024-09-17T00:23:33.741 INFO:journalctl@ceph.mon.a.smithi002.stdout: block_cache: 0x55bc595611f0 2024-09-17T00:23:33.741 INFO:journalctl@ceph.mon.a.smithi002.stdout: block_cache_name: BinnedLRUCache 2024-09-17T00:23:33.741 INFO:journalctl@ceph.mon.a.smithi002.stdout: block_cache_options: 2024-09-17T00:23:33.741 INFO:journalctl@ceph.mon.a.smithi002.stdout: capacity : 536870912 2024-09-17T00:23:33.741 INFO:journalctl@ceph.mon.a.smithi002.stdout: num_shard_bits : 4 2024-09-17T00:23:33.742 INFO:journalctl@ceph.mon.a.smithi002.stdout: strict_capacity_limit : 0 2024-09-17T00:23:33.742 INFO:journalctl@ceph.mon.a.smithi002.stdout: high_pri_pool_ratio: 0.000 2024-09-17T00:23:33.742 INFO:journalctl@ceph.mon.a.smithi002.stdout: block_cache_compressed: (nil) 2024-09-17T00:23:33.742 INFO:journalctl@ceph.mon.a.smithi002.stdout: persistent_cache: (nil) 2024-09-17T00:23:33.742 INFO:journalctl@ceph.mon.a.smithi002.stdout: block_size: 4096 2024-09-17T00:23:33.742 INFO:journalctl@ceph.mon.a.smithi002.stdout: block_size_deviation: 10 2024-09-17T00:23:33.742 INFO:journalctl@ceph.mon.a.smithi002.stdout: block_restart_interval: 16 2024-09-17T00:23:33.742 INFO:journalctl@ceph.mon.a.smithi002.stdout: index_block_restart_interval: 1 2024-09-17T00:23:33.742 INFO:journalctl@ceph.mon.a.smithi002.stdout: metadata_block_size: 4096 2024-09-17T00:23:33.742 INFO:journalctl@ceph.mon.a.smithi002.stdout: partition_filters: 0 2024-09-17T00:23:33.742 INFO:journalctl@ceph.mon.a.smithi002.stdout: use_delta_encoding: 1 2024-09-17T00:23:33.743 INFO:journalctl@ceph.mon.a.smithi002.stdout: filter_policy: bloomfilter 2024-09-17T00:23:33.743 INFO:journalctl@ceph.mon.a.smithi002.stdout: whole_key_filtering: 1 2024-09-17T00:23:33.743 INFO:journalctl@ceph.mon.a.smithi002.stdout: verify_compression: 0 2024-09-17T00:23:33.743 INFO:journalctl@ceph.mon.a.smithi002.stdout: read_amp_bytes_per_bit: 0 2024-09-17T00:23:33.743 INFO:journalctl@ceph.mon.a.smithi002.stdout: format_version: 5 2024-09-17T00:23:33.743 INFO:journalctl@ceph.mon.a.smithi002.stdout: enable_index_compression: 1 2024-09-17T00:23:33.743 INFO:journalctl@ceph.mon.a.smithi002.stdout: block_align: 0 2024-09-17T00:23:33.743 INFO:journalctl@ceph.mon.a.smithi002.stdout: max_auto_readahead_size: 262144 2024-09-17T00:23:33.743 INFO:journalctl@ceph.mon.a.smithi002.stdout: prepopulate_block_cache: 0 2024-09-17T00:23:33.743 INFO:journalctl@ceph.mon.a.smithi002.stdout: initial_auto_readahead_size: 8192 2024-09-17T00:23:33.743 INFO:journalctl@ceph.mon.a.smithi002.stdout: num_file_reads_for_auto_readahead: 2 2024-09-17T00:23:33.743 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.write_buffer_size: 33554432 2024-09-17T00:23:33.743 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.max_write_buffer_number: 2 2024-09-17T00:23:33.744 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.compression: NoCompression 2024-09-17T00:23:33.744 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.bottommost_compression: Disabled 2024-09-17T00:23:33.744 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.prefix_extractor: nullptr 2024-09-17T00:23:33.744 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-17T00:23:33.744 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.num_levels: 7 2024-09-17T00:23:33.744 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-17T00:23:33.744 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-17T00:23:33.744 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-17T00:23:33.744 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-17T00:23:33.745 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-17T00:23:33.745 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-17T00:23:33.745 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-17T00:23:33.745 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-17T00:23:33.745 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-17T00:23:33.745 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-17T00:23:33.745 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-09-17T00:23:33.745 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-09-17T00:23:33.745 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-17T00:23:33.745 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.compression_opts.level: 32767 2024-09-17T00:23:33.745 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.compression_opts.strategy: 0 2024-09-17T00:23:33.746 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-17T00:23:33.746 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-17T00:23:33.746 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-09-17T00:23:33.746 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-17T00:23:33.746 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.compression_opts.enabled: false 2024-09-17T00:23:33.746 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-09-17T00:23:33.747 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-17T00:23:33.747 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-17T00:23:33.747 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-17T00:23:33.747 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.target_file_size_base: 67108864 2024-09-17T00:23:33.747 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-17T00:23:33.747 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-17T00:23:33.747 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-17T00:23:33.748 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-17T00:23:33.748 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-17T00:23:33.748 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-17T00:23:33.748 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-17T00:23:33.748 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-17T00:23:33.748 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-17T00:23:33.748 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-17T00:23:33.748 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-17T00:23:33.748 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-17T00:23:33.748 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-17T00:23:33.749 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-09-17T00:23:33.749 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.arena_block_size: 1048576 2024-09-17T00:23:33.749 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-17T00:23:33.749 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-17T00:23:33.749 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.disable_auto_compactions: 0 2024-09-17T00:23:33.749 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-17T00:23:33.749 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-17T00:23:33.749 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-17T00:23:33.749 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-17T00:23:33.749 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-17T00:23:33.749 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-17T00:23:33.749 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-17T00:23:33.749 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-17T00:23:33.750 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-17T00:23:33.750 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-17T00:23:33.750 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-17T00:23:33.750 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.inplace_update_support: 0 2024-09-17T00:23:33.750 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-17T00:23:33.750 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-17T00:23:33.750 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-17T00:23:33.750 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-17T00:23:33.750 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.bloom_locality: 0 2024-09-17T00:23:33.750 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.max_successive_merges: 0 2024-09-17T00:23:33.750 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-17T00:23:33.751 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.paranoid_file_checks: 0 2024-09-17T00:23:33.751 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.force_consistency_checks: 1 2024-09-17T00:23:33.751 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.report_bg_io_stats: 0 2024-09-17T00:23:33.751 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.ttl: 2592000 2024-09-17T00:23:33.751 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-17T00:23:33.751 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-09-17T00:23:33.751 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-09-17T00:23:33.751 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.enable_blob_files: false 2024-09-17T00:23:33.751 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.min_blob_size: 0 2024-09-17T00:23:33.751 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.blob_file_size: 268435456 2024-09-17T00:23:33.751 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-17T00:23:33.751 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-17T00:23:33.751 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-17T00:23:33.751 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-09-17T00:23:33.751 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-09-17T00:23:33.752 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.blob_file_starting_level: 0 2024-09-17T00:23:33.752 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-09-17T00:23:33.752 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 30.sst 2024-09-17T00:23:33.752 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-09-17T00:23:33.752 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000015 succeeded,manifest_file_number is 15, next_file_number is 32, last_sequence is 12105, log_number is 28,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-09-17T00:23:33.752 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 28 2024-09-17T00:23:33.752 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 45cae19e-03ac-46b2-a1b3-52301bc67af3 2024-09-17T00:23:33.752 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726532613457323, "job": 1, "event": "recovery_started", "wal_files": [28]} 2024-09-17T00:23:33.752 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #28 mode 2 2024-09-17T00:23:33.752 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726532613467580, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 33, "file_size": 3249752, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 12106, "largest_seqno": 13670, "table_properties": {"data_size": 3241987, "index_size": 4927, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 1797, "raw_key_size": 16501, "raw_average_key_size": 23, "raw_value_size": 3226659, "raw_average_value_size": 4642, "num_data_blocks": 223, "num_entries": 695, "num_filter_entries": 695, "num_deletions": 2, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1726532613, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "45cae19e-03ac-46b2-a1b3-52301bc67af3", "db_session_id": "PE7BPYGZLXW1ES7SVVVL", "orig_file_number": 33, "seqno_to_time_mapping": "N/A"}} 2024-09-17T00:23:33.752 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726532613467709, "job": 1, "event": "recovery_finished"} 2024-09-17T00:23:33.752 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: [db/version_set.cc:5047] Creating manifest 35 2024-09-17T00:23:33.753 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-09-17T00:23:33.753 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000028.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-17T00:23:33.753 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55bc5958ee00 2024-09-17T00:23:33.753 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: DB pointer 0x55bc5968a000 2024-09-17T00:23:33.753 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-09-17T00:23:33.753 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-09-17T00:23:33.753 INFO:journalctl@ceph.mon.a.smithi002.stdout: ** DB Stats ** 2024-09-17T00:23:33.753 INFO:journalctl@ceph.mon.a.smithi002.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-17T00:23:33.753 INFO:journalctl@ceph.mon.a.smithi002.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-17T00:23:33.753 INFO:journalctl@ceph.mon.a.smithi002.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-17T00:23:33.753 INFO:journalctl@ceph.mon.a.smithi002.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-17T00:23:33.753 INFO:journalctl@ceph.mon.a.smithi002.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-17T00:23:33.753 INFO:journalctl@ceph.mon.a.smithi002.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-17T00:23:33.753 INFO:journalctl@ceph.mon.a.smithi002.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-17T00:23:33.754 INFO:journalctl@ceph.mon.a.smithi002.stdout: 2024-09-17T00:23:33.754 INFO:journalctl@ceph.mon.a.smithi002.stdout: ** Compaction Stats [default] ** 2024-09-17T00:23:33.754 INFO:journalctl@ceph.mon.a.smithi002.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-17T00:23:33.754 INFO:journalctl@ceph.mon.a.smithi002.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-09-17T00:23:33.754 INFO:journalctl@ceph.mon.a.smithi002.stdout: L0 1/0 3.10 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 599.9 0.01 0.00 1 0.005 0 0 0.0 0.0 2024-09-17T00:23:33.754 INFO:journalctl@ceph.mon.a.smithi002.stdout: L6 1/0 8.01 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-09-17T00:23:33.754 INFO:journalctl@ceph.mon.a.smithi002.stdout: Sum 2/0 11.11 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 599.9 0.01 0.00 1 0.005 0 0 0.0 0.0 2024-09-17T00:23:33.754 INFO:journalctl@ceph.mon.a.smithi002.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 599.9 0.01 0.00 1 0.005 0 0 0.0 0.0 2024-09-17T00:23:33.754 INFO:journalctl@ceph.mon.a.smithi002.stdout: 2024-09-17T00:23:33.754 INFO:journalctl@ceph.mon.a.smithi002.stdout: ** Compaction Stats [default] ** 2024-09-17T00:23:33.755 INFO:journalctl@ceph.mon.a.smithi002.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-17T00:23:33.755 INFO:journalctl@ceph.mon.a.smithi002.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-17T00:23:33.755 INFO:journalctl@ceph.mon.a.smithi002.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 599.9 0.01 0.00 1 0.005 0 0 0.0 0.0 2024-09-17T00:23:33.755 INFO:journalctl@ceph.mon.a.smithi002.stdout: 2024-09-17T00:23:33.755 INFO:journalctl@ceph.mon.a.smithi002.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-09-17T00:23:33.755 INFO:journalctl@ceph.mon.a.smithi002.stdout: 2024-09-17T00:23:33.755 INFO:journalctl@ceph.mon.a.smithi002.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-17T00:23:33.755 INFO:journalctl@ceph.mon.a.smithi002.stdout: Flush(GB): cumulative 0.003, interval 0.003 2024-09-17T00:23:33.755 INFO:journalctl@ceph.mon.a.smithi002.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-17T00:23:33.755 INFO:journalctl@ceph.mon.a.smithi002.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-17T00:23:33.755 INFO:journalctl@ceph.mon.a.smithi002.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-17T00:23:33.755 INFO:journalctl@ceph.mon.a.smithi002.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-17T00:23:33.755 INFO:journalctl@ceph.mon.a.smithi002.stdout: Cumulative compaction: 0.00 GB write, 241.09 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-17T00:23:33.755 INFO:journalctl@ceph.mon.a.smithi002.stdout: Interval compaction: 0.00 GB write, 241.09 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-17T00:23:33.756 INFO:journalctl@ceph.mon.a.smithi002.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-17T00:23:33.756 INFO:journalctl@ceph.mon.a.smithi002.stdout: Block cache BinnedLRUCache@0x55bc595611f0#2 capacity: 512.00 MB usage: 7.11 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2024-09-17T00:23:33.756 INFO:journalctl@ceph.mon.a.smithi002.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,2.03 KB,0.00038743%) IndexBlock(1,5.08 KB,0.000968575%) Misc(1,0.00 KB,0%) 2024-09-17T00:23:33.756 INFO:journalctl@ceph.mon.a.smithi002.stdout: 2024-09-17T00:23:33.756 INFO:journalctl@ceph.mon.a.smithi002.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-17T00:23:33.756 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: starting mon.a rank 0 at public addrs [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] at bind addrs [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 3aad27a2-7489-11ef-bceb-c7b262605968 2024-09-17T00:23:33.756 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: mon.a@-1(???) e3 preinit fsid 3aad27a2-7489-11ef-bceb-c7b262605968 2024-09-17T00:23:33.756 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: mon.a@-1(???).mds e1 new map 2024-09-17T00:23:33.756 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: mon.a@-1(???).mds e1 print_map 2024-09-17T00:23:33.756 INFO:journalctl@ceph.mon.a.smithi002.stdout: e1 2024-09-17T00:23:33.756 INFO:journalctl@ceph.mon.a.smithi002.stdout: btime 1970-01-01T00:00:00:000000+0000 2024-09-17T00:23:33.756 INFO:journalctl@ceph.mon.a.smithi002.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-17T00:23:33.757 INFO:journalctl@ceph.mon.a.smithi002.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-09-17T00:23:33.757 INFO:journalctl@ceph.mon.a.smithi002.stdout: legacy client fscid: -1 2024-09-17T00:23:33.757 INFO:journalctl@ceph.mon.a.smithi002.stdout: 2024-09-17T00:23:33.757 INFO:journalctl@ceph.mon.a.smithi002.stdout: No filesystems configured 2024-09-17T00:23:33.757 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: mon.a@-1(???).osd e105 crush map has features 3314933000854323200, adjusting msgr requires 2024-09-17T00:23:33.757 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: mon.a@-1(???).osd e105 crush map has features 432629239337189376, adjusting msgr requires 2024-09-17T00:23:33.757 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: mon.a@-1(???).osd e105 crush map has features 432629239337189376, adjusting msgr requires 2024-09-17T00:23:33.757 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: mon.a@-1(???).osd e105 crush map has features 432629239337189376, adjusting msgr requires 2024-09-17T00:23:33.757 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 ceph-mon[63804]: mon.a@-1(???).paxosservice(auth 1..22) refresh upgraded, format 0 -> 3 2024-09-17T00:23:33.757 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 bash[63785]: a035def53ef68b141bb6821552dcaaad986ad917acefceef91af915e462c773c 2024-09-17T00:23:33.757 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:33 smithi002 systemd[1]: Started Ceph mon.a for 3aad27a2-7489-11ef-bceb-c7b262605968. 2024-09-17T00:23:33.967 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:23:33 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[52757]: ts=2024-09-17T00:23:33.506Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes name=CephNodeDiskspaceWarning index=4 msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi002\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi002\", job=\"node\", machine=\"x86_64\", nodename=\"smithi002\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"3aad27a2-7489-11ef-bceb-c7b262605968\", domainname=\"(none)\", instance=\"smithi002\", job=\"node\", machine=\"x86_64\", nodename=\"smithi002\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-17T00:23:36.856 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:23:36 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[53240]: ignoring --setuser ceph since I am not root 2024-09-17T00:23:36.856 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:23:36 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[53240]: ignoring --setgroup ceph since I am not root 2024-09-17T00:23:36.856 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:23:36 smithi099 ceph-mgr[53244]: -- 172.21.15.99:0/2671848976 <== mon.2 v2:172.21.15.99:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x564c45ecb0e0 con 0x564c45ea9400 2024-09-17T00:23:36.856 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:23:36 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[53240]: 2024-09-17T00:23:36.654+0000 7fb582b2b040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-17T00:23:36.856 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:23:36 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[53240]: 2024-09-17T00:23:36.711+0000 7fb582b2b040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-17T00:23:36.856 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:23:36 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[53240]: 2024-09-17T00:23:36.858+0000 7fb582b2b040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-17T00:23:36.857 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:36 smithi099 ceph-mon[24594]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:23:36.857 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:36 smithi099 ceph-mon[24594]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:23:36.857 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:36 smithi099 ceph-mon[24594]: mon.a calling monitor election 2024-09-17T00:23:36.857 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:36 smithi099 ceph-mon[24594]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-09-17T00:23:36.857 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:36 smithi099 ceph-mon[24594]: monmap epoch 3 2024-09-17T00:23:36.857 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:36 smithi099 ceph-mon[24594]: fsid 3aad27a2-7489-11ef-bceb-c7b262605968 2024-09-17T00:23:36.857 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:36 smithi099 ceph-mon[24594]: last_changed 2024-09-17T00:12:24.385107+0000 2024-09-17T00:23:36.857 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:36 smithi099 ceph-mon[24594]: created 2024-09-17T00:10:42.458448+0000 2024-09-17T00:23:36.857 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:36 smithi099 ceph-mon[24594]: min_mon_release 17 (quincy) 2024-09-17T00:23:36.857 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:36 smithi099 ceph-mon[24594]: election_strategy: 1 2024-09-17T00:23:36.857 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:36 smithi099 ceph-mon[24594]: 0: [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] mon.a 2024-09-17T00:23:36.858 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:36 smithi099 ceph-mon[24594]: 1: [v2:172.21.15.2:3301/0,v1:172.21.15.2:6790/0] mon.c 2024-09-17T00:23:36.858 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:36 smithi099 ceph-mon[24594]: 2: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.b 2024-09-17T00:23:36.858 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:36 smithi099 ceph-mon[24594]: fsmap 2024-09-17T00:23:36.858 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:36 smithi099 ceph-mon[24594]: osdmap e105: 8 total, 8 up, 8 in 2024-09-17T00:23:36.858 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:36 smithi099 ceph-mon[24594]: mgrmap e33: y(active, since 40s), standbys: x 2024-09-17T00:23:36.858 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:36 smithi099 ceph-mon[24594]: overall HEALTH_OK 2024-09-17T00:23:36.858 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:36 smithi099 ceph-mon[24594]: from='mgr.15333 ' entity='' 2024-09-17T00:23:36.858 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:36 smithi099 ceph-mon[24594]: mgrmap e34: y(active, since 40s), standbys: x 2024-09-17T00:23:36.880 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:36 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: ignoring --setuser ceph since I am not root 2024-09-17T00:23:36.880 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:36 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: ignoring --setgroup ceph since I am not root 2024-09-17T00:23:36.880 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:36 smithi002 ceph-mgr[59220]: -- 172.21.15.2:0/1886681462 <== mon.1 v2:172.21.15.2:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55c1c3ebb0e0 con 0x55c1c3e98800 2024-09-17T00:23:36.880 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:36 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: 2024-09-17T00:23:36.667+0000 7f18511be040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-17T00:23:36.880 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:36 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: 2024-09-17T00:23:36.727+0000 7f18511be040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-17T00:23:36.880 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:36 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: 2024-09-17T00:23:36.882+0000 7f18511be040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-17T00:23:36.880 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:36 smithi002 ceph-mon[28203]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:23:36.881 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:36 smithi002 ceph-mon[28203]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:23:36.881 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:36 smithi002 ceph-mon[28203]: mon.a calling monitor election 2024-09-17T00:23:36.881 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:36 smithi002 ceph-mon[28203]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-09-17T00:23:36.881 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:36 smithi002 ceph-mon[28203]: monmap epoch 3 2024-09-17T00:23:36.881 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:36 smithi002 ceph-mon[28203]: fsid 3aad27a2-7489-11ef-bceb-c7b262605968 2024-09-17T00:23:36.881 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:36 smithi002 ceph-mon[28203]: last_changed 2024-09-17T00:12:24.385107+0000 2024-09-17T00:23:36.881 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:36 smithi002 ceph-mon[28203]: created 2024-09-17T00:10:42.458448+0000 2024-09-17T00:23:36.881 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:36 smithi002 ceph-mon[28203]: min_mon_release 17 (quincy) 2024-09-17T00:23:36.881 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:36 smithi002 ceph-mon[28203]: election_strategy: 1 2024-09-17T00:23:36.881 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:36 smithi002 ceph-mon[28203]: 0: [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] mon.a 2024-09-17T00:23:36.881 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:36 smithi002 ceph-mon[28203]: 1: [v2:172.21.15.2:3301/0,v1:172.21.15.2:6790/0] mon.c 2024-09-17T00:23:36.881 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:36 smithi002 ceph-mon[28203]: 2: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.b 2024-09-17T00:23:36.882 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:36 smithi002 ceph-mon[28203]: fsmap 2024-09-17T00:23:36.882 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:36 smithi002 ceph-mon[28203]: osdmap e105: 8 total, 8 up, 8 in 2024-09-17T00:23:36.882 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:36 smithi002 ceph-mon[28203]: mgrmap e33: y(active, since 40s), standbys: x 2024-09-17T00:23:36.882 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:36 smithi002 ceph-mon[28203]: overall HEALTH_OK 2024-09-17T00:23:36.882 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:36 smithi002 ceph-mon[28203]: from='mgr.15333 ' entity='' 2024-09-17T00:23:36.882 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:36 smithi002 ceph-mon[28203]: mgrmap e34: y(active, since 40s), standbys: x 2024-09-17T00:23:36.882 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:36 smithi002 ceph-mon[63804]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:23:36.882 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:36 smithi002 ceph-mon[63804]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:23:36.882 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:36 smithi002 ceph-mon[63804]: mon.a calling monitor election 2024-09-17T00:23:36.882 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:36 smithi002 ceph-mon[63804]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-09-17T00:23:36.882 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:36 smithi002 ceph-mon[63804]: monmap epoch 3 2024-09-17T00:23:36.882 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:36 smithi002 ceph-mon[63804]: fsid 3aad27a2-7489-11ef-bceb-c7b262605968 2024-09-17T00:23:36.883 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:36 smithi002 ceph-mon[63804]: last_changed 2024-09-17T00:12:24.385107+0000 2024-09-17T00:23:36.883 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:36 smithi002 ceph-mon[63804]: created 2024-09-17T00:10:42.458448+0000 2024-09-17T00:23:36.883 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:36 smithi002 ceph-mon[63804]: min_mon_release 17 (quincy) 2024-09-17T00:23:36.883 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:36 smithi002 ceph-mon[63804]: election_strategy: 1 2024-09-17T00:23:36.883 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:36 smithi002 ceph-mon[63804]: 0: [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] mon.a 2024-09-17T00:23:36.883 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:36 smithi002 ceph-mon[63804]: 1: [v2:172.21.15.2:3301/0,v1:172.21.15.2:6790/0] mon.c 2024-09-17T00:23:36.883 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:36 smithi002 ceph-mon[63804]: 2: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.b 2024-09-17T00:23:36.883 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:36 smithi002 ceph-mon[63804]: fsmap 2024-09-17T00:23:36.883 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:36 smithi002 ceph-mon[63804]: osdmap e105: 8 total, 8 up, 8 in 2024-09-17T00:23:36.883 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:36 smithi002 ceph-mon[63804]: mgrmap e33: y(active, since 40s), standbys: x 2024-09-17T00:23:36.883 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:36 smithi002 ceph-mon[63804]: overall HEALTH_OK 2024-09-17T00:23:36.883 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:36 smithi002 ceph-mon[63804]: from='mgr.15333 ' entity='' 2024-09-17T00:23:36.883 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:36 smithi002 ceph-mon[63804]: mgrmap e34: y(active, since 40s), standbys: x 2024-09-17T00:23:37.214 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:23:36 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[53240]: 2024-09-17T00:23:36.926+0000 7fb582b2b040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-17T00:23:37.214 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:23:37 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[53240]: 2024-09-17T00:23:37.073+0000 7fb582b2b040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-17T00:23:37.229 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:36 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: 2024-09-17T00:23:36.952+0000 7f18511be040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-17T00:23:37.229 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:37 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: 2024-09-17T00:23:37.105+0000 7f18511be040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-17T00:23:37.466 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:23:37 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[53240]: 2024-09-17T00:23:37.223+0000 7fb582b2b040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-17T00:23:37.581 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:37 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: 2024-09-17T00:23:37.253+0000 7f18511be040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-17T00:23:37.786 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:23:37 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[53240]: 2024-09-17T00:23:37.537+0000 7fb582b2b040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-17T00:23:37.787 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:23:37 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[53240]: 2024-09-17T00:23:37.598+0000 7fb582b2b040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-17T00:23:37.787 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:23:37 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[53240]: 2024-09-17T00:23:37.789+0000 7fb582b2b040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-17T00:23:37.840 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:37 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: 2024-09-17T00:23:37.583+0000 7f18511be040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-17T00:23:37.841 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:37 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: 2024-09-17T00:23:37.647+0000 7f18511be040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-17T00:23:38.124 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:37 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: 2024-09-17T00:23:37.843+0000 7f18511be040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-17T00:23:38.806 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:38 smithi002 ceph-mon[63804]: mgrmap e35: y(active, since 41s), standbys: x 2024-09-17T00:23:38.806 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:38 smithi002 ceph-mon[28203]: mgrmap e35: y(active, since 41s), standbys: x 2024-09-17T00:23:38.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:38 smithi099 ceph-mon[24594]: mgrmap e35: y(active, since 41s), standbys: x 2024-09-17T00:23:38.967 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:23:38 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[53240]: 2024-09-17T00:23:38.673+0000 7fb582b2b040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-17T00:23:38.967 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:23:38 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[53240]: 2024-09-17T00:23:38.741+0000 7fb582b2b040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-17T00:23:39.105 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:38 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: 2024-09-17T00:23:38.807+0000 7f18511be040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-17T00:23:39.105 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:38 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: 2024-09-17T00:23:38.889+0000 7f18511be040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-17T00:23:39.712 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:23:39 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[53240]: 2024-09-17T00:23:39.454+0000 7fb582b2b040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-17T00:23:39.898 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:23:39 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[53240]: 2024-09-17T00:23:39.572+0000 7fb582b2b040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-17T00:23:39.899 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:39 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: 2024-09-17T00:23:39.678+0000 7f18511be040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-17T00:23:39.966 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:23:39 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[53240]: 2024-09-17T00:23:39.714+0000 7fb582b2b040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-17T00:23:39.967 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:23:39 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[53240]: 2024-09-17T00:23:39.777+0000 7fb582b2b040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-17T00:23:39.967 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:23:39 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[53240]: 2024-09-17T00:23:39.929+0000 7fb582b2b040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-17T00:23:40.192 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:39 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: 2024-09-17T00:23:39.805+0000 7f18511be040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-17T00:23:40.192 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:39 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: 2024-09-17T00:23:39.962+0000 7f18511be040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-17T00:23:40.192 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:40 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: 2024-09-17T00:23:40.033+0000 7f18511be040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-17T00:23:40.287 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:23:40 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[53240]: 2024-09-17T00:23:40.001+0000 7fb582b2b040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-17T00:23:40.287 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:23:40 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[53240]: 2024-09-17T00:23:40.062+0000 7fb582b2b040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-17T00:23:40.287 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:23:40 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[53240]: 2024-09-17T00:23:40.223+0000 7fb582b2b040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-17T00:23:40.287 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:23:40 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[53240]: 2024-09-17T00:23:40.290+0000 7fb582b2b040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-17T00:23:40.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:40 smithi002 ceph-mon[28203]: from='client.? 172.21.15.2:0/1257824255' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-17T00:23:40.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:40 smithi002 ceph-mon[63804]: from='client.? 172.21.15.2:0/1257824255' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-17T00:23:40.480 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:40 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: 2024-09-17T00:23:40.194+0000 7f18511be040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-17T00:23:40.480 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:40 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: 2024-09-17T00:23:40.266+0000 7f18511be040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-17T00:23:40.480 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:40 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: 2024-09-17T00:23:40.330+0000 7f18511be040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-17T00:23:40.555 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:40 smithi099 ceph-mon[24594]: from='client.? 172.21.15.2:0/1257824255' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-17T00:23:40.555 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:23:40 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[53240]: 2024-09-17T00:23:40.356+0000 7fb582b2b040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-17T00:23:40.555 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:23:40 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[53240]: 2024-09-17T00:23:40.439+0000 7fb582b2b040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-17T00:23:40.555 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:23:40 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[53240]: 2024-09-17T00:23:40.499+0000 7fb582b2b040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-17T00:23:40.785 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:40 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: 2024-09-17T00:23:40.500+0000 7f18511be040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-17T00:23:40.785 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:40 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: 2024-09-17T00:23:40.569+0000 7f18511be040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-17T00:23:40.785 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:40 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: 2024-09-17T00:23:40.638+0000 7f18511be040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-17T00:23:40.785 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:40 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: 2024-09-17T00:23:40.725+0000 7f18511be040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-17T00:23:40.785 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:40 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: 2024-09-17T00:23:40.787+0000 7f18511be040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-17T00:23:40.967 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:23:40 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[53240]: 2024-09-17T00:23:40.557+0000 7fb582b2b040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-17T00:23:41.229 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:40 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: 2024-09-17T00:23:40.849+0000 7f18511be040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-17T00:23:41.682 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:41 smithi002 ceph-mon[63804]: Standby manager daemon x restarted 2024-09-17T00:23:41.682 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:41 smithi002 ceph-mon[63804]: Standby manager daemon x started 2024-09-17T00:23:41.682 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:41 smithi002 ceph-mon[63804]: from='mgr.? 172.21.15.99:0/1577422662' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-17T00:23:41.683 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:41 smithi002 ceph-mon[63804]: from='mgr.? 172.21.15.99:0/1577422662' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-17T00:23:41.683 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:41 smithi002 ceph-mon[63804]: from='mgr.? 172.21.15.99:0/1577422662' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-17T00:23:41.683 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:41 smithi002 ceph-mon[63804]: from='mgr.? 172.21.15.99:0/1577422662' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-17T00:23:41.683 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:41 smithi002 ceph-mon[28203]: Standby manager daemon x restarted 2024-09-17T00:23:41.683 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:41 smithi002 ceph-mon[28203]: Standby manager daemon x started 2024-09-17T00:23:41.683 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:41 smithi002 ceph-mon[28203]: from='mgr.? 172.21.15.99:0/1577422662' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-17T00:23:41.683 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:41 smithi002 ceph-mon[28203]: from='mgr.? 172.21.15.99:0/1577422662' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-17T00:23:41.683 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:41 smithi002 ceph-mon[28203]: from='mgr.? 172.21.15.99:0/1577422662' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-17T00:23:41.683 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:41 smithi002 ceph-mon[28203]: from='mgr.? 172.21.15.99:0/1577422662' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-17T00:23:41.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:41 smithi099 ceph-mon[24594]: Standby manager daemon x restarted 2024-09-17T00:23:41.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:41 smithi099 ceph-mon[24594]: Standby manager daemon x started 2024-09-17T00:23:41.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:41 smithi099 ceph-mon[24594]: from='mgr.? 172.21.15.99:0/1577422662' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-17T00:23:41.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:41 smithi099 ceph-mon[24594]: from='mgr.? 172.21.15.99:0/1577422662' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-17T00:23:41.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:41 smithi099 ceph-mon[24594]: from='mgr.? 172.21.15.99:0/1577422662' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-17T00:23:41.718 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:41 smithi099 ceph-mon[24594]: from='mgr.? 172.21.15.99:0/1577422662' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-17T00:23:41.718 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:23:41 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[53240]: 2024-09-17T00:23:41.351+0000 7fb582b2b040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-17T00:23:41.718 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:23:41 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[53240]: [17/Sep/2024:00:23:41] ENGINE Bus STARTING 2024-09-17T00:23:41.718 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:23:41 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[53240]: CherryPy Checker: 2024-09-17T00:23:41.718 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:23:41 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[53240]: The Application mounted at '' has an empty config. 2024-09-17T00:23:41.718 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:23:41 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[53240]: 2024-09-17T00:23:41.718 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:23:41 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[53240]: [17/Sep/2024:00:23:41] ENGINE Serving on http://:::9283 2024-09-17T00:23:41.718 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:23:41 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x[53240]: [17/Sep/2024:00:23:41] ENGINE Bus STARTED 2024-09-17T00:23:41.979 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:41 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: 2024-09-17T00:23:41.684+0000 7f18511be040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-17T00:23:41.979 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:41 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: [17/Sep/2024:00:23:41] ENGINE Bus STARTING 2024-09-17T00:23:41.979 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:41 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: CherryPy Checker: 2024-09-17T00:23:41.979 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:41 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: The Application mounted at '' has an empty config. 2024-09-17T00:23:41.979 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:41 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: 2024-09-17T00:23:41.979 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:41 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: [17/Sep/2024:00:23:41] ENGINE Serving on http://:::9283 2024-09-17T00:23:41.979 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:41 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: [17/Sep/2024:00:23:41] ENGINE Bus STARTED 2024-09-17T00:23:42.705 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:42 smithi002 ceph-mon[28203]: mgrmap e36: y(active, since 45s), standbys: x 2024-09-17T00:23:42.705 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:42 smithi002 ceph-mon[28203]: Active manager daemon y restarted 2024-09-17T00:23:42.705 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:42 smithi002 ceph-mon[28203]: Activating manager daemon y 2024-09-17T00:23:42.705 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:42 smithi002 ceph-mon[28203]: osdmap e106: 8 total, 8 up, 8 in 2024-09-17T00:23:42.705 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:42 smithi002 ceph-mon[28203]: mgrmap e37: y(active, starting, since 0.00763415s), standbys: x 2024-09-17T00:23:42.705 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:42 smithi002 ceph-mon[28203]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T00:23:42.705 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:42 smithi002 ceph-mon[28203]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T00:23:42.705 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:42 smithi002 ceph-mon[28203]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T00:23:42.706 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:42 smithi002 ceph-mon[28203]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-17T00:23:42.706 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:42 smithi002 ceph-mon[28203]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-17T00:23:42.706 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:42 smithi002 ceph-mon[28203]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-17T00:23:42.706 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:42 smithi002 ceph-mon[28203]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-17T00:23:42.706 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:42 smithi002 ceph-mon[28203]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-17T00:23:42.706 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:42 smithi002 ceph-mon[28203]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T00:23:42.706 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:42 smithi002 ceph-mon[28203]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T00:23:42.706 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:42 smithi002 ceph-mon[28203]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T00:23:42.706 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:42 smithi002 ceph-mon[28203]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T00:23:42.706 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:42 smithi002 ceph-mon[28203]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T00:23:42.706 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:42 smithi002 ceph-mon[28203]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T00:23:42.706 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:42 smithi002 ceph-mon[28203]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T00:23:42.707 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:42 smithi002 ceph-mon[28203]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T00:23:42.707 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:42 smithi002 ceph-mon[28203]: Manager daemon y is now available 2024-09-17T00:23:42.707 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:42 smithi002 ceph-mon[28203]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T00:23:42.707 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:42 smithi002 ceph-mon[28203]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T00:23:42.707 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:42 smithi002 ceph-mon[28203]: from='mgr.25256 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T00:23:42.707 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:42 smithi002 ceph-mon[28203]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T00:23:42.707 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:42 smithi002 ceph-mon[28203]: from='mgr.25256 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T00:23:42.707 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:42 smithi002 ceph-mon[63804]: mgrmap e36: y(active, since 45s), standbys: x 2024-09-17T00:23:42.707 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:42 smithi002 ceph-mon[63804]: Active manager daemon y restarted 2024-09-17T00:23:42.707 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:42 smithi002 ceph-mon[63804]: Activating manager daemon y 2024-09-17T00:23:42.707 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:42 smithi002 ceph-mon[63804]: osdmap e106: 8 total, 8 up, 8 in 2024-09-17T00:23:42.708 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:42 smithi002 ceph-mon[63804]: mgrmap e37: y(active, starting, since 0.00763415s), standbys: x 2024-09-17T00:23:42.708 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:42 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T00:23:42.708 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:42 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T00:23:42.708 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:42 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T00:23:42.708 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:42 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-17T00:23:42.708 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:42 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-17T00:23:42.708 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:42 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-17T00:23:42.708 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:42 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-17T00:23:42.708 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:42 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-17T00:23:42.708 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:42 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T00:23:42.708 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:42 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T00:23:42.708 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:42 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T00:23:42.709 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:42 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T00:23:42.709 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:42 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T00:23:42.709 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:42 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T00:23:42.709 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:42 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T00:23:42.709 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:42 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T00:23:42.709 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:42 smithi002 ceph-mon[63804]: Manager daemon y is now available 2024-09-17T00:23:42.709 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:42 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T00:23:42.709 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:42 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T00:23:42.709 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:42 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T00:23:42.709 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:42 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T00:23:42.709 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:42 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T00:23:42.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:42 smithi099 ceph-mon[24594]: mgrmap e36: y(active, since 45s), standbys: x 2024-09-17T00:23:42.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:42 smithi099 ceph-mon[24594]: Active manager daemon y restarted 2024-09-17T00:23:42.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:42 smithi099 ceph-mon[24594]: Activating manager daemon y 2024-09-17T00:23:42.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:42 smithi099 ceph-mon[24594]: osdmap e106: 8 total, 8 up, 8 in 2024-09-17T00:23:42.718 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:42 smithi099 ceph-mon[24594]: mgrmap e37: y(active, starting, since 0.00763415s), standbys: x 2024-09-17T00:23:42.718 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:42 smithi099 ceph-mon[24594]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T00:23:42.718 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:42 smithi099 ceph-mon[24594]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T00:23:42.718 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:42 smithi099 ceph-mon[24594]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T00:23:42.718 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:42 smithi099 ceph-mon[24594]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-17T00:23:42.718 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:42 smithi099 ceph-mon[24594]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-17T00:23:42.718 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:42 smithi099 ceph-mon[24594]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-17T00:23:42.718 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:42 smithi099 ceph-mon[24594]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-17T00:23:42.718 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:42 smithi099 ceph-mon[24594]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-17T00:23:42.718 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:42 smithi099 ceph-mon[24594]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T00:23:42.718 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:42 smithi099 ceph-mon[24594]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T00:23:42.719 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:42 smithi099 ceph-mon[24594]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T00:23:42.719 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:42 smithi099 ceph-mon[24594]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T00:23:42.719 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:42 smithi099 ceph-mon[24594]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T00:23:42.719 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:42 smithi099 ceph-mon[24594]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T00:23:42.719 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:42 smithi099 ceph-mon[24594]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T00:23:42.719 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:42 smithi099 ceph-mon[24594]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T00:23:42.719 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:42 smithi099 ceph-mon[24594]: Manager daemon y is now available 2024-09-17T00:23:42.719 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:42 smithi099 ceph-mon[24594]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T00:23:42.719 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:42 smithi099 ceph-mon[24594]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T00:23:42.719 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:42 smithi099 ceph-mon[24594]: from='mgr.25256 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-17T00:23:42.719 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:42 smithi099 ceph-mon[24594]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T00:23:42.719 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:42 smithi099 ceph-mon[24594]: from='mgr.25256 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-17T00:23:42.720 INFO:teuthology.orchestra.run.smithi002.stdout:true 2024-09-17T00:23:42.979 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:42 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: 2024-09-17T00:23:42.706+0000 7f17efec7640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-09-17T00:23:43.469 INFO:teuthology.orchestra.run.smithi002.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-17T00:23:43.470 INFO:teuthology.orchestra.run.smithi002.stdout:alertmanager.a smithi002 *:9093,9094 running (93s) 40s ago 6m 16.9M - 0.25.0 c8568f914cd2 a21e13b3d677 2024-09-17T00:23:43.470 INFO:teuthology.orchestra.run.smithi002.stdout:grafana.a smithi099 *:3000 running (66s) 21s ago 6m 43.0M - 8.3.5 dad864ee21e9 bafb824a3c08 2024-09-17T00:23:43.470 INFO:teuthology.orchestra.run.smithi002.stdout:iscsi.foo.smithi002.vmouem smithi002 running (48s) 40s ago 5m 45.9M - 3.5 e1d6a67b021e cdd8a2092ce4 2024-09-17T00:23:43.470 INFO:teuthology.orchestra.run.smithi002.stdout:mgr.x smithi099 *:8443,9283,8765 running (26s) 21s ago 11m 64.4M - 19.1.1-330-gc2ddf972 ebb0615149de 6aa2ccef9ca0 2024-09-17T00:23:43.470 INFO:teuthology.orchestra.run.smithi002.stdout:mgr.y smithi002 *:8443,9283,8765 running (61s) 40s ago 12m 474M - 19.1.1-330-gc2ddf972 ebb0615149de 8838897e8449 2024-09-17T00:23:43.470 INFO:teuthology.orchestra.run.smithi002.stdout:mon.a smithi002 running (12m) 40s ago 12m 58.1M 2048M 17.2.0 e1d6a67b021e a68823df625f 2024-09-17T00:23:43.470 INFO:teuthology.orchestra.run.smithi002.stdout:mon.b smithi099 running (11m) 21s ago 11m 47.9M 2048M 17.2.0 e1d6a67b021e c6d2af715187 2024-09-17T00:23:43.470 INFO:teuthology.orchestra.run.smithi002.stdout:mon.c smithi002 running (11m) 40s ago 11m 43.3M 2048M 17.2.0 e1d6a67b021e 77b3d2e14899 2024-09-17T00:23:43.470 INFO:teuthology.orchestra.run.smithi002.stdout:node-exporter.a smithi002 *:9100 running (102s) 40s ago 7m 8002k - 1.5.0 0da6a335fe13 1030ae01bb32 2024-09-17T00:23:43.470 INFO:teuthology.orchestra.run.smithi002.stdout:node-exporter.b smithi099 *:9100 running (71s) 21s ago 7m 8548k - 1.5.0 0da6a335fe13 01fcd35dca8a 2024-09-17T00:23:43.470 INFO:teuthology.orchestra.run.smithi002.stdout:osd.0 smithi002 running (10m) 40s ago 10m 52.4M 2737M 17.2.0 e1d6a67b021e a0ad0fb7aa34 2024-09-17T00:23:43.471 INFO:teuthology.orchestra.run.smithi002.stdout:osd.1 smithi002 running (10m) 40s ago 10m 55.5M 2737M 17.2.0 e1d6a67b021e 7cb62c311c7d 2024-09-17T00:23:43.471 INFO:teuthology.orchestra.run.smithi002.stdout:osd.2 smithi002 running (9m) 40s ago 9m 51.8M 2737M 17.2.0 e1d6a67b021e eafddacb955e 2024-09-17T00:23:43.471 INFO:teuthology.orchestra.run.smithi002.stdout:osd.3 smithi002 running (9m) 40s ago 9m 55.8M 2737M 17.2.0 e1d6a67b021e fd99d3764412 2024-09-17T00:23:43.471 INFO:teuthology.orchestra.run.smithi002.stdout:osd.4 smithi099 running (9m) 21s ago 9m 55.6M 2992M 17.2.0 e1d6a67b021e 0af1827b658d 2024-09-17T00:23:43.471 INFO:teuthology.orchestra.run.smithi002.stdout:osd.5 smithi099 running (8m) 21s ago 8m 54.5M 2992M 17.2.0 e1d6a67b021e 4f7ef3aefc58 2024-09-17T00:23:43.471 INFO:teuthology.orchestra.run.smithi002.stdout:osd.6 smithi099 running (8m) 21s ago 8m 50.0M 2992M 17.2.0 e1d6a67b021e 3213f5e6952a 2024-09-17T00:23:43.471 INFO:teuthology.orchestra.run.smithi002.stdout:osd.7 smithi099 running (7m) 21s ago 7m 56.6M 2992M 17.2.0 e1d6a67b021e 63be3a1ed0a1 2024-09-17T00:23:43.471 INFO:teuthology.orchestra.run.smithi002.stdout:prometheus.a smithi099 *:9095 running (31s) 21s ago 7m 43.0M - 2.43.0 a07b618ecd1d bf32c2ae11dc 2024-09-17T00:23:43.471 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.foo.smithi002.uwdhcs smithi002 *:8000 running (5m) 40s ago 5m 74.1M - 17.2.0 e1d6a67b021e 9c35813f459a 2024-09-17T00:23:43.471 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.foo.smithi099.utzfap smithi099 *:8000 running (5m) 21s ago 5m 75.1M - 17.2.0 e1d6a67b021e 34ec989160a6 2024-09-17T00:23:43.471 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.smpl.smithi002.rqnxii smithi002 *:80 running (5m) 40s ago 5m 72.6M - 17.2.0 e1d6a67b021e 0e180d0afb99 2024-09-17T00:23:43.471 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.smpl.smithi099.nzvkpm smithi099 *:80 running (5m) 21s ago 5m 75.7M - 17.2.0 e1d6a67b021e 6a068e82ac72 2024-09-17T00:23:43.717 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:23:43 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[52757]: ts=2024-09-17T00:23:43.505Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes name=CephNodeDiskspaceWarning index=4 msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi002\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi002\", job=\"node\", machine=\"x86_64\", nodename=\"smithi002\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"3aad27a2-7489-11ef-bceb-c7b262605968\", domainname=\"(none)\", instance=\"smithi002\", job=\"node\", machine=\"x86_64\", nodename=\"smithi002\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-17T00:23:43.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:43 smithi099 ceph-mon[24594]: mgrmap e38: y(active, since 1.01027s), standbys: x 2024-09-17T00:23:43.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:43 smithi099 ceph-mon[24594]: from='client.25393 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:23:43.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:43 smithi099 ceph-mon[24594]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:23:43.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:43 smithi099 ceph-mon[24594]: from='client.34158 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:23:43.718 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:43 smithi099 ceph-mon[24594]: from='client.25405 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:23:43.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:43 smithi002 ceph-mon[28203]: mgrmap e38: y(active, since 1.01027s), standbys: x 2024-09-17T00:23:43.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:43 smithi002 ceph-mon[28203]: from='client.25393 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:23:43.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:43 smithi002 ceph-mon[28203]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:23:43.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:43 smithi002 ceph-mon[28203]: from='client.34158 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:23:43.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:43 smithi002 ceph-mon[28203]: from='client.25405 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:23:43.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:43 smithi002 ceph-mon[63804]: mgrmap e38: y(active, since 1.01027s), standbys: x 2024-09-17T00:23:43.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:43 smithi002 ceph-mon[63804]: from='client.25393 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:23:43.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:43 smithi002 ceph-mon[63804]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:23:43.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:43 smithi002 ceph-mon[63804]: from='client.34158 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:23:43.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:43 smithi002 ceph-mon[63804]: from='client.25405 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:23:43.909 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-09-17T00:23:43.909 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": { 2024-09-17T00:23:43.909 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2, 2024-09-17T00:23:43.909 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 19.1.1-330-gc2ddf972 (c2ddf9721ef254645201cdbabd9c8f49c30a586f) squid (stable)": 1 2024-09-17T00:23:43.909 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-09-17T00:23:43.909 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": { 2024-09-17T00:23:43.909 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 19.1.1-330-gc2ddf972 (c2ddf9721ef254645201cdbabd9c8f49c30a586f) squid (stable)": 2 2024-09-17T00:23:43.910 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-09-17T00:23:43.910 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": { 2024-09-17T00:23:43.910 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-09-17T00:23:43.910 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-09-17T00:23:43.910 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw": { 2024-09-17T00:23:43.910 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-09-17T00:23:43.910 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-09-17T00:23:43.910 INFO:teuthology.orchestra.run.smithi002.stdout: "overall": { 2024-09-17T00:23:43.910 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 14, 2024-09-17T00:23:43.910 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 19.1.1-330-gc2ddf972 (c2ddf9721ef254645201cdbabd9c8f49c30a586f) squid (stable)": 3 2024-09-17T00:23:43.910 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-09-17T00:23:43.910 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-09-17T00:23:44.276 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-09-17T00:23:44.277 INFO:teuthology.orchestra.run.smithi002.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f", 2024-09-17T00:23:44.277 INFO:teuthology.orchestra.run.smithi002.stdout: "in_progress": true, 2024-09-17T00:23:44.277 INFO:teuthology.orchestra.run.smithi002.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-17T00:23:44.277 INFO:teuthology.orchestra.run.smithi002.stdout: "services_complete": [ 2024-09-17T00:23:44.277 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr" 2024-09-17T00:23:44.277 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-09-17T00:23:44.277 INFO:teuthology.orchestra.run.smithi002.stdout: "progress": "2/23 daemons upgraded", 2024-09-17T00:23:44.277 INFO:teuthology.orchestra.run.smithi002.stdout: "message": "", 2024-09-17T00:23:44.277 INFO:teuthology.orchestra.run.smithi002.stdout: "is_paused": false 2024-09-17T00:23:44.277 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-09-17T00:23:44.697 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:44 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: 2024-09-17T00:23:44.483+0000 7f17a41a8640 -1 log_channel(cephadm) log [ERR] : [17/Sep/2024:00:23:44] ENGINE Error in HTTPServer.serve 2024-09-17T00:23:44.698 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:44 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: Traceback (most recent call last): 2024-09-17T00:23:44.698 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:44 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-17T00:23:44.698 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:44 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: self._connections.run(self.expiration_interval) 2024-09-17T00:23:44.698 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:44 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-17T00:23:44.698 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:44 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: self._run(expiration_interval) 2024-09-17T00:23:44.698 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:44 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-17T00:23:44.698 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:44 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: new_conn = self._from_server_socket(self.server.socket) 2024-09-17T00:23:44.698 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:44 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-17T00:23:44.699 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:44 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-17T00:23:44.699 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:44 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-17T00:23:44.699 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:44 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: s = self.context.wrap_socket( 2024-09-17T00:23:44.699 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:44 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-17T00:23:44.699 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:44 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: return self.sslsocket_class._create( 2024-09-17T00:23:44.699 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:44 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-17T00:23:44.699 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:44 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: self.do_handshake() 2024-09-17T00:23:44.699 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:44 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-17T00:23:44.699 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:44 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: self._sslobj.do_handshake() 2024-09-17T00:23:44.699 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:44 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-17T00:23:44.699 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:44 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: 2024-09-17T00:23:44.739 INFO:teuthology.orchestra.run.smithi002.stdout:HEALTH_OK 2024-09-17T00:23:44.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:44 smithi099 ceph-mon[24594]: mgrmap e39: y(active, since 2s), standbys: x 2024-09-17T00:23:44.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:44 smithi099 ceph-mon[24594]: from='client.? 172.21.15.2:0/1921978387' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:23:44.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:44 smithi099 ceph-mon[24594]: [17/Sep/2024:00:23:44] ENGINE Bus STARTING 2024-09-17T00:23:44.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:44 smithi099 ceph-mon[24594]: from='client.34182 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:23:44.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:44 smithi099 ceph-mon[24594]: [17/Sep/2024:00:23:44] ENGINE Serving on http://172.21.15.2:8765 2024-09-17T00:23:44.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:44 smithi099 ceph-mon[24594]: [17/Sep/2024:00:23:44] ENGINE Serving on https://172.21.15.2:7150 2024-09-17T00:23:44.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:44 smithi099 ceph-mon[24594]: [17/Sep/2024:00:23:44] ENGINE Bus STARTED 2024-09-17T00:23:44.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:44 smithi099 ceph-mon[24594]: [17/Sep/2024:00:23:44] ENGINE Error in HTTPServer.serve 2024-09-17T00:23:44.968 INFO:journalctl@ceph.mon.b.smithi099.stdout: Traceback (most recent call last): 2024-09-17T00:23:44.968 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-17T00:23:44.968 INFO:journalctl@ceph.mon.b.smithi099.stdout: self._connections.run(self.expiration_interval) 2024-09-17T00:23:44.968 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-17T00:23:44.968 INFO:journalctl@ceph.mon.b.smithi099.stdout: self._run(expiration_interval) 2024-09-17T00:23:44.968 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-17T00:23:44.968 INFO:journalctl@ceph.mon.b.smithi099.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-17T00:23:44.968 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-17T00:23:44.968 INFO:journalctl@ceph.mon.b.smithi099.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-17T00:23:44.968 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-17T00:23:44.968 INFO:journalctl@ceph.mon.b.smithi099.stdout: s = self.context.wrap_socket( 2024-09-17T00:23:44.968 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-17T00:23:44.968 INFO:journalctl@ceph.mon.b.smithi099.stdout: return self.sslsocket_class._create( 2024-09-17T00:23:44.969 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-17T00:23:44.969 INFO:journalctl@ceph.mon.b.smithi099.stdout: self.do_handshake() 2024-09-17T00:23:44.969 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-17T00:23:44.969 INFO:journalctl@ceph.mon.b.smithi099.stdout: self._sslobj.do_handshake() 2024-09-17T00:23:44.969 INFO:journalctl@ceph.mon.b.smithi099.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-17T00:23:44.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:44 smithi002 ceph-mon[63804]: mgrmap e39: y(active, since 2s), standbys: x 2024-09-17T00:23:44.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:44 smithi002 ceph-mon[63804]: from='client.? 172.21.15.2:0/1921978387' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:23:44.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:44 smithi002 ceph-mon[63804]: [17/Sep/2024:00:23:44] ENGINE Bus STARTING 2024-09-17T00:23:44.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:44 smithi002 ceph-mon[63804]: from='client.34182 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:23:44.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:44 smithi002 ceph-mon[63804]: [17/Sep/2024:00:23:44] ENGINE Serving on http://172.21.15.2:8765 2024-09-17T00:23:44.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:44 smithi002 ceph-mon[63804]: [17/Sep/2024:00:23:44] ENGINE Serving on https://172.21.15.2:7150 2024-09-17T00:23:44.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:44 smithi002 ceph-mon[63804]: [17/Sep/2024:00:23:44] ENGINE Bus STARTED 2024-09-17T00:23:44.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:44 smithi002 ceph-mon[63804]: [17/Sep/2024:00:23:44] ENGINE Error in HTTPServer.serve 2024-09-17T00:23:44.979 INFO:journalctl@ceph.mon.a.smithi002.stdout: Traceback (most recent call last): 2024-09-17T00:23:44.980 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-17T00:23:44.980 INFO:journalctl@ceph.mon.a.smithi002.stdout: self._connections.run(self.expiration_interval) 2024-09-17T00:23:44.980 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-17T00:23:44.980 INFO:journalctl@ceph.mon.a.smithi002.stdout: self._run(expiration_interval) 2024-09-17T00:23:44.980 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-17T00:23:44.980 INFO:journalctl@ceph.mon.a.smithi002.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-17T00:23:44.980 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-17T00:23:44.980 INFO:journalctl@ceph.mon.a.smithi002.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-17T00:23:44.980 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-17T00:23:44.980 INFO:journalctl@ceph.mon.a.smithi002.stdout: s = self.context.wrap_socket( 2024-09-17T00:23:44.980 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-17T00:23:44.980 INFO:journalctl@ceph.mon.a.smithi002.stdout: return self.sslsocket_class._create( 2024-09-17T00:23:44.980 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-17T00:23:44.981 INFO:journalctl@ceph.mon.a.smithi002.stdout: self.do_handshake() 2024-09-17T00:23:44.981 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-17T00:23:44.981 INFO:journalctl@ceph.mon.a.smithi002.stdout: self._sslobj.do_handshake() 2024-09-17T00:23:44.981 INFO:journalctl@ceph.mon.a.smithi002.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-17T00:23:44.981 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:44 smithi002 ceph-mon[28203]: mgrmap e39: y(active, since 2s), standbys: x 2024-09-17T00:23:44.981 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:44 smithi002 ceph-mon[28203]: from='client.? 172.21.15.2:0/1921978387' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:23:44.981 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:44 smithi002 ceph-mon[28203]: [17/Sep/2024:00:23:44] ENGINE Bus STARTING 2024-09-17T00:23:44.981 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:44 smithi002 ceph-mon[28203]: from='client.34182 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:23:44.981 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:44 smithi002 ceph-mon[28203]: [17/Sep/2024:00:23:44] ENGINE Serving on http://172.21.15.2:8765 2024-09-17T00:23:44.981 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:44 smithi002 ceph-mon[28203]: [17/Sep/2024:00:23:44] ENGINE Serving on https://172.21.15.2:7150 2024-09-17T00:23:44.981 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:44 smithi002 ceph-mon[28203]: [17/Sep/2024:00:23:44] ENGINE Bus STARTED 2024-09-17T00:23:44.981 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:44 smithi002 ceph-mon[28203]: [17/Sep/2024:00:23:44] ENGINE Error in HTTPServer.serve 2024-09-17T00:23:44.982 INFO:journalctl@ceph.mon.c.smithi002.stdout: Traceback (most recent call last): 2024-09-17T00:23:44.982 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-17T00:23:44.982 INFO:journalctl@ceph.mon.c.smithi002.stdout: self._connections.run(self.expiration_interval) 2024-09-17T00:23:44.982 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-17T00:23:44.982 INFO:journalctl@ceph.mon.c.smithi002.stdout: self._run(expiration_interval) 2024-09-17T00:23:44.982 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-17T00:23:44.982 INFO:journalctl@ceph.mon.c.smithi002.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-17T00:23:44.982 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-17T00:23:44.982 INFO:journalctl@ceph.mon.c.smithi002.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-17T00:23:44.982 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-17T00:23:44.982 INFO:journalctl@ceph.mon.c.smithi002.stdout: s = self.context.wrap_socket( 2024-09-17T00:23:44.982 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-17T00:23:44.982 INFO:journalctl@ceph.mon.c.smithi002.stdout: return self.sslsocket_class._create( 2024-09-17T00:23:44.983 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-17T00:23:44.983 INFO:journalctl@ceph.mon.c.smithi002.stdout: self.do_handshake() 2024-09-17T00:23:44.983 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-17T00:23:44.983 INFO:journalctl@ceph.mon.c.smithi002.stdout: self._sslobj.do_handshake() 2024-09-17T00:23:44.983 INFO:journalctl@ceph.mon.c.smithi002.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-17T00:23:45.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:45 smithi002 ceph-mon[63804]: from='client.? 172.21.15.2:0/2023945912' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T00:23:45.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:45 smithi002 ceph-mon[28203]: from='client.? 172.21.15.2:0/2023945912' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T00:23:45.953 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:45 smithi099 ceph-mon[24594]: from='client.? 172.21.15.2:0/2023945912' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T00:23:46.957 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:46 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: ::ffff:172.21.15.99 - - [17/Sep/2024:00:23:46] "GET /metrics HTTP/1.1" 200 35360 "" "Prometheus/2.43.0" 2024-09-17T00:23:46.957 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:46 smithi002 ceph-mon[28203]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:23:46.958 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:46 smithi002 ceph-mon[28203]: mgrmap e40: y(active, since 4s), standbys: x 2024-09-17T00:23:46.958 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:46 smithi002 ceph-mon[28203]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:23:46.958 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:46 smithi002 ceph-mon[28203]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:23:46.958 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:46 smithi002 ceph-mon[63804]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:23:46.958 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:46 smithi002 ceph-mon[63804]: mgrmap e40: y(active, since 4s), standbys: x 2024-09-17T00:23:46.958 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:46 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:23:46.958 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:46 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:23:46.966 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:46 smithi099 ceph-mon[24594]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:23:46.966 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:46 smithi099 ceph-mon[24594]: mgrmap e40: y(active, since 4s), standbys: x 2024-09-17T00:23:46.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:46 smithi099 ceph-mon[24594]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:23:46.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:46 smithi099 ceph-mon[24594]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:23:48.105 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:47 smithi099 ceph-mon[24594]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:23:48.106 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:47 smithi099 ceph-mon[24594]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:23:48.106 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:47 smithi099 ceph-mon[24594]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:23:48.106 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:47 smithi099 ceph-mon[24594]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:23:48.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:47 smithi002 ceph-mon[28203]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:23:48.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:47 smithi002 ceph-mon[28203]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:23:48.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:47 smithi002 ceph-mon[28203]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:23:48.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:47 smithi002 ceph-mon[28203]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:23:48.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:47 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:23:48.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:47 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:23:48.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:47 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:23:48.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:47 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:23:48.717 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:23:48 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[52757]: ts=2024-09-17T00:23:48.659Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd name=CephOSDFlapping index=13 msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.2\", device_class=\"ssd\", hostname=\"smithi002\", instance=\"172.21.15.99:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.2\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"3aad27a2-7489-11ef-bceb-c7b262605968\", cluster_addr=\"172.21.15.2\", device_class=\"ssd\", hostname=\"smithi002\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.2\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-17T00:23:49.466 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:49 smithi099 ceph-mon[24594]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:23:49.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:49 smithi099 ceph-mon[24594]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:23:49.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:49 smithi099 ceph-mon[24594]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:23:49.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:49 smithi002 ceph-mon[63804]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:23:49.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:49 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:23:49.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:49 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:23:49.480 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:49 smithi002 ceph-mon[28203]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:23:49.480 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:49 smithi002 ceph-mon[28203]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:23:49.480 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:49 smithi002 ceph-mon[28203]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:23:51.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:51 smithi099 ceph-mon[24594]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-09-17T00:23:51.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:51 smithi099 ceph-mon[24594]: from='client.25271 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:23:51.468 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:51 smithi099 ceph-mon[24594]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:23:51.468 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:51 smithi099 ceph-mon[24594]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:23:51.468 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:51 smithi099 ceph-mon[24594]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:23:51.468 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:51 smithi099 ceph-mon[24594]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:23:51.468 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:51 smithi099 ceph-mon[24594]: Updating smithi002:/etc/ceph/ceph.conf 2024-09-17T00:23:51.468 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:51 smithi099 ceph-mon[24594]: Updating smithi099:/etc/ceph/ceph.conf 2024-09-17T00:23:51.468 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:51 smithi099 ceph-mon[24594]: Updating smithi099:/var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-17T00:23:51.469 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:51 smithi099 ceph-mon[24594]: Updating smithi002:/var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-17T00:23:51.469 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:51 smithi099 ceph-mon[24594]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:23:51.469 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:51 smithi099 ceph-mon[24594]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:23:51.469 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:51 smithi099 ceph-mon[24594]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:23:51.469 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:51 smithi099 ceph-mon[24594]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:23:51.469 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:51 smithi099 ceph-mon[24594]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:23:51.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:51 smithi002 ceph-mon[28203]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-09-17T00:23:51.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:51 smithi002 ceph-mon[28203]: from='client.25271 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:23:51.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:51 smithi002 ceph-mon[28203]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:23:51.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:51 smithi002 ceph-mon[28203]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:23:51.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:51 smithi002 ceph-mon[28203]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:23:51.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:51 smithi002 ceph-mon[28203]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:23:51.480 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:51 smithi002 ceph-mon[28203]: Updating smithi002:/etc/ceph/ceph.conf 2024-09-17T00:23:51.480 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:51 smithi002 ceph-mon[28203]: Updating smithi099:/etc/ceph/ceph.conf 2024-09-17T00:23:51.480 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:51 smithi002 ceph-mon[28203]: Updating smithi099:/var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-17T00:23:51.480 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:51 smithi002 ceph-mon[28203]: Updating smithi002:/var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-17T00:23:51.480 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:51 smithi002 ceph-mon[28203]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:23:51.480 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:51 smithi002 ceph-mon[28203]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:23:51.480 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:51 smithi002 ceph-mon[28203]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:23:51.480 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:51 smithi002 ceph-mon[28203]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:23:51.480 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:51 smithi002 ceph-mon[28203]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:23:51.481 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:51 smithi002 ceph-mon[63804]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-09-17T00:23:51.481 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:51 smithi002 ceph-mon[63804]: from='client.25271 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:23:51.481 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:51 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:23:51.481 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:51 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:23:51.481 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:51 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:23:51.481 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:51 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:23:51.481 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:51 smithi002 ceph-mon[63804]: Updating smithi002:/etc/ceph/ceph.conf 2024-09-17T00:23:51.481 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:51 smithi002 ceph-mon[63804]: Updating smithi099:/etc/ceph/ceph.conf 2024-09-17T00:23:51.481 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:51 smithi002 ceph-mon[63804]: Updating smithi099:/var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-17T00:23:51.481 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:51 smithi002 ceph-mon[63804]: Updating smithi002:/var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/config/ceph.conf 2024-09-17T00:23:51.481 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:51 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:23:51.482 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:51 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:23:51.482 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:51 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:23:51.482 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:51 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:23:51.482 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:51 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:23:52.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:52 smithi099 ceph-mon[24594]: Updating smithi099:/etc/ceph/ceph.client.admin.keyring 2024-09-17T00:23:52.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:52 smithi099 ceph-mon[24594]: Updating smithi002:/etc/ceph/ceph.client.admin.keyring 2024-09-17T00:23:52.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:52 smithi099 ceph-mon[24594]: Updating smithi099:/var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-17T00:23:52.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:52 smithi099 ceph-mon[24594]: Updating smithi002:/var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-17T00:23:52.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:52 smithi099 ceph-mon[24594]: Reconfiguring iscsi.foo.smithi002.vmouem (dependencies changed)... 2024-09-17T00:23:52.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:52 smithi099 ceph-mon[24594]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi002.vmouem", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-17T00:23:52.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:52 smithi099 ceph-mon[24594]: from='mgr.25256 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi002.vmouem", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-17T00:23:52.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:52 smithi099 ceph-mon[24594]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:23:52.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:52 smithi099 ceph-mon[24594]: Reconfiguring daemon iscsi.foo.smithi002.vmouem on smithi002 2024-09-17T00:23:52.470 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:52 smithi002 ceph-mon[28203]: Updating smithi099:/etc/ceph/ceph.client.admin.keyring 2024-09-17T00:23:52.471 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:52 smithi002 ceph-mon[28203]: Updating smithi002:/etc/ceph/ceph.client.admin.keyring 2024-09-17T00:23:52.471 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:52 smithi002 ceph-mon[28203]: Updating smithi099:/var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-17T00:23:52.471 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:52 smithi002 ceph-mon[28203]: Updating smithi002:/var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-17T00:23:52.471 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:52 smithi002 ceph-mon[28203]: Reconfiguring iscsi.foo.smithi002.vmouem (dependencies changed)... 2024-09-17T00:23:52.471 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:52 smithi002 ceph-mon[28203]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi002.vmouem", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-17T00:23:52.471 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:52 smithi002 ceph-mon[28203]: from='mgr.25256 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi002.vmouem", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-17T00:23:52.471 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:52 smithi002 ceph-mon[28203]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:23:52.471 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:52 smithi002 ceph-mon[28203]: Reconfiguring daemon iscsi.foo.smithi002.vmouem on smithi002 2024-09-17T00:23:52.471 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:52 smithi002 ceph-mon[63804]: Updating smithi099:/etc/ceph/ceph.client.admin.keyring 2024-09-17T00:23:52.472 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:52 smithi002 ceph-mon[63804]: Updating smithi002:/etc/ceph/ceph.client.admin.keyring 2024-09-17T00:23:52.472 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:52 smithi002 ceph-mon[63804]: Updating smithi099:/var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-17T00:23:52.472 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:52 smithi002 ceph-mon[63804]: Updating smithi002:/var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/config/ceph.client.admin.keyring 2024-09-17T00:23:52.472 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:52 smithi002 ceph-mon[63804]: Reconfiguring iscsi.foo.smithi002.vmouem (dependencies changed)... 2024-09-17T00:23:52.472 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:52 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi002.vmouem", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-17T00:23:52.472 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:52 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi002.vmouem", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-17T00:23:52.472 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:52 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:23:52.472 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:52 smithi002 ceph-mon[63804]: Reconfiguring daemon iscsi.foo.smithi002.vmouem on smithi002 2024-09-17T00:23:53.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:53 smithi002 ceph-mon[28203]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-09-17T00:23:53.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:53 smithi002 ceph-mon[63804]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-09-17T00:23:53.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:53 smithi099 ceph-mon[24594]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-09-17T00:23:53.967 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:23:53 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[52757]: ts=2024-09-17T00:23:53.506Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes name=CephNodeDiskspaceWarning index=4 msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi002\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi002\", job=\"node\", machine=\"x86_64\", nodename=\"smithi002\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"3aad27a2-7489-11ef-bceb-c7b262605968\", domainname=\"(none)\", instance=\"smithi002\", job=\"node\", machine=\"x86_64\", nodename=\"smithi002\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-17T00:23:54.479 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:54 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: 2024-09-17T00:23:54.227+0000 7f17e1e2b640 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi002.vmouem ... 2024-09-17T00:23:54.479 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:54 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: Creating ceph-iscsi config... 2024-09-17T00:23:54.479 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:54 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/iscsi-gateway.cfg 2024-09-17T00:23:54.480 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:54 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/tcmu-runner-entrypoint.sh 2024-09-17T00:23:54.480 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:54 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: Non-zero exit code 1 from systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem 2024-09-17T00:23:54.480 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:54 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: systemctl: stderr Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:23:54.480 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:54 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: systemctl: stderr See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:23:54.480 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:54 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: Traceback (most recent call last): 2024-09-17T00:23:54.480 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:54 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-17T00:23:54.480 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:54 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: return _run_code(code, main_globals, None, 2024-09-17T00:23:54.480 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:54 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-17T00:23:54.480 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:54 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: exec(code, run_globals) 2024-09-17T00:23:54.480 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:54 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5579, in 2024-09-17T00:23:54.480 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:54 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5567, in main 2024-09-17T00:23:54.481 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:54 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3049, in command_deploy_from 2024-09-17T00:23:54.481 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:54 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3084, in _common_deploy 2024-09-17T00:23:54.481 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:54 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3104, in _deploy_daemon_container 2024-09-17T00:23:54.481 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:54 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 1124, in deploy_daemon 2024-09-17T00:23:54.481 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:54 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/tmp/tmpnroy8d0w.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-17T00:23:54.481 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:54 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: RuntimeError: Failed command: systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem: Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:23:54.481 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:54 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:23:54.481 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:54 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: Traceback (most recent call last): 2024-09-17T00:23:54.482 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:54 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1134, in _check_daemons 2024-09-17T00:23:54.482 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:54 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: self.mgr._daemon_action(daemon_spec, action=action) 2024-09-17T00:23:54.482 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:54 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2473, in _daemon_action 2024-09-17T00:23:54.482 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:54 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: return self.wait_async( 2024-09-17T00:23:54.482 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:54 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/usr/share/ceph/mgr/cephadm/module.py", line 796, in wait_async 2024-09-17T00:23:54.482 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:54 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: return self.event_loop.get_result(coro, timeout) 2024-09-17T00:23:54.482 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:54 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-09-17T00:23:54.482 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:54 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: return future.result(timeout) 2024-09-17T00:23:54.482 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:54 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/lib64/python3.9/concurrent/futures/_base.py", line 446, in result 2024-09-17T00:23:54.482 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:54 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: return self.__get_result() 2024-09-17T00:23:54.482 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:54 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/lib64/python3.9/concurrent/futures/_base.py", line 391, in __get_result 2024-09-17T00:23:54.482 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:54 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: raise self._exception 2024-09-17T00:23:54.483 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:54 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1370, in _create_daemon 2024-09-17T00:23:54.483 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:54 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: out, err, code = await self._run_cephadm( 2024-09-17T00:23:54.483 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:54 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1713, in _run_cephadm 2024-09-17T00:23:54.483 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:54 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: raise OrchestratorError( 2024-09-17T00:23:54.483 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:54 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi002.vmouem ... 2024-09-17T00:23:54.483 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:54 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: Creating ceph-iscsi config... 2024-09-17T00:23:54.483 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:54 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/iscsi-gateway.cfg 2024-09-17T00:23:54.483 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:54 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/tcmu-runner-entrypoint.sh 2024-09-17T00:23:54.483 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:54 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: Non-zero exit code 1 from systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem 2024-09-17T00:23:54.483 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:54 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: systemctl: stderr Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:23:54.484 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:54 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: systemctl: stderr See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:23:54.484 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:54 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: Traceback (most recent call last): 2024-09-17T00:23:54.484 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:54 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-17T00:23:54.484 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:54 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: return _run_code(code, main_globals, None, 2024-09-17T00:23:54.484 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:54 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-17T00:23:54.484 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:54 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: exec(code, run_globals) 2024-09-17T00:23:54.484 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:54 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5579, in 2024-09-17T00:23:54.484 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:54 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5567, in main 2024-09-17T00:23:54.484 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:54 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3049, in command_deploy_from 2024-09-17T00:23:54.484 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:54 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3084, in _common_deploy 2024-09-17T00:23:54.484 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:54 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3104, in _deploy_daemon_container 2024-09-17T00:23:54.484 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:54 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 1124, in deploy_daemon 2024-09-17T00:23:54.484 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:54 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/tmp/tmpnroy8d0w.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-17T00:23:54.484 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:54 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: RuntimeError: Failed command: systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem: Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:23:54.485 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:54 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:23:55.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:55 smithi002 ceph-mon[28203]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-17T00:23:55.230 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:55 smithi002 ceph-mon[28203]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi002.vmouem ... 2024-09-17T00:23:55.230 INFO:journalctl@ceph.mon.c.smithi002.stdout: Creating ceph-iscsi config... 2024-09-17T00:23:55.230 INFO:journalctl@ceph.mon.c.smithi002.stdout: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/iscsi-gateway.cfg 2024-09-17T00:23:55.230 INFO:journalctl@ceph.mon.c.smithi002.stdout: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/tcmu-runner-entrypoint.sh 2024-09-17T00:23:55.230 INFO:journalctl@ceph.mon.c.smithi002.stdout: Non-zero exit code 1 from systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem 2024-09-17T00:23:55.230 INFO:journalctl@ceph.mon.c.smithi002.stdout: systemctl: stderr Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:23:55.230 INFO:journalctl@ceph.mon.c.smithi002.stdout: systemctl: stderr See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:23:55.230 INFO:journalctl@ceph.mon.c.smithi002.stdout: Traceback (most recent call last): 2024-09-17T00:23:55.230 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-17T00:23:55.230 INFO:journalctl@ceph.mon.c.smithi002.stdout: return _run_code(code, main_globals, None, 2024-09-17T00:23:55.230 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-17T00:23:55.230 INFO:journalctl@ceph.mon.c.smithi002.stdout: exec(code, run_globals) 2024-09-17T00:23:55.230 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5579, in 2024-09-17T00:23:55.231 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5567, in main 2024-09-17T00:23:55.231 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3049, in command_deploy_from 2024-09-17T00:23:55.231 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3084, in _common_deploy 2024-09-17T00:23:55.231 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3104, in _deploy_daemon_container 2024-09-17T00:23:55.231 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 1124, in deploy_daemon 2024-09-17T00:23:55.231 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-17T00:23:55.231 INFO:journalctl@ceph.mon.c.smithi002.stdout: RuntimeError: Failed command: systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem: Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:23:55.231 INFO:journalctl@ceph.mon.c.smithi002.stdout: See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:23:55.231 INFO:journalctl@ceph.mon.c.smithi002.stdout: Traceback (most recent call last): 2024-09-17T00:23:55.231 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1134, in _check_daemons 2024-09-17T00:23:55.231 INFO:journalctl@ceph.mon.c.smithi002.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-09-17T00:23:55.231 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2473, in _daemon_action 2024-09-17T00:23:55.231 INFO:journalctl@ceph.mon.c.smithi002.stdout: return self.wait_async( 2024-09-17T00:23:55.232 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 796, in wait_async 2024-09-17T00:23:55.232 INFO:journalctl@ceph.mon.c.smithi002.stdout: return self.event_loop.get_result(coro, timeout) 2024-09-17T00:23:55.232 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-09-17T00:23:55.232 INFO:journalctl@ceph.mon.c.smithi002.stdout: return future.result(timeout) 2024-09-17T00:23:55.232 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/lib64/python3.9/concurrent/futures/_base.py", line 446, in result 2024-09-17T00:23:55.232 INFO:journalctl@ceph.mon.c.smithi002.stdout: return self.__get_result() 2024-09-17T00:23:55.232 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/lib64/python3.9/concurrent/futures/_base.py", line 391, in __get_result 2024-09-17T00:23:55.232 INFO:journalctl@ceph.mon.c.smithi002.stdout: raise self._exception 2024-09-17T00:23:55.232 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1370, in _create_daemon 2024-09-17T00:23:55.232 INFO:journalctl@ceph.mon.c.smithi002.stdout: out, err, code = await self._run_cephadm( 2024-09-17T00:23:55.232 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1713, in _run_cephadm 2024-09-17T00:23:55.232 INFO:journalctl@ceph.mon.c.smithi002.stdout: raise OrchestratorError( 2024-09-17T00:23:55.232 INFO:journalctl@ceph.mon.c.smithi002.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi002.vmouem ... 2024-09-17T00:23:55.233 INFO:journalctl@ceph.mon.c.smithi002.stdout: Creating ceph-iscsi config... 2024-09-17T00:23:55.233 INFO:journalctl@ceph.mon.c.smithi002.stdout: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/iscsi-gateway.cfg 2024-09-17T00:23:55.233 INFO:journalctl@ceph.mon.c.smithi002.stdout: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/tcmu-runner-entrypoint.sh 2024-09-17T00:23:55.233 INFO:journalctl@ceph.mon.c.smithi002.stdout: Non-zero exit code 1 from systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem 2024-09-17T00:23:55.233 INFO:journalctl@ceph.mon.c.smithi002.stdout: systemctl: stderr Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:23:55.233 INFO:journalctl@ceph.mon.c.smithi002.stdout: systemctl: stderr See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:23:55.233 INFO:journalctl@ceph.mon.c.smithi002.stdout: Traceback (most recent call last): 2024-09-17T00:23:55.233 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-17T00:23:55.233 INFO:journalctl@ceph.mon.c.smithi002.stdout: return _run_code(code, main_globals, None, 2024-09-17T00:23:55.233 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-17T00:23:55.233 INFO:journalctl@ceph.mon.c.smithi002.stdout: exec(code, run_globals) 2024-09-17T00:23:55.233 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5579, in 2024-09-17T00:23:55.233 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5567, in main 2024-09-17T00:23:55.233 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3049, in command_deploy_from 2024-09-17T00:23:55.234 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3084, in _common_deploy 2024-09-17T00:23:55.234 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3104, in _deploy_daemon_container 2024-09-17T00:23:55.234 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 1124, in deploy_daemon 2024-09-17T00:23:55.234 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-17T00:23:55.234 INFO:journalctl@ceph.mon.c.smithi002.stdout: RuntimeError: Failed command: systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem: Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:23:55.234 INFO:journalctl@ceph.mon.c.smithi002.stdout: See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:23:55.234 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:55 smithi002 ceph-mon[63804]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-17T00:23:55.234 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:55 smithi002 ceph-mon[63804]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi002.vmouem ... 2024-09-17T00:23:55.234 INFO:journalctl@ceph.mon.a.smithi002.stdout: Creating ceph-iscsi config... 2024-09-17T00:23:55.234 INFO:journalctl@ceph.mon.a.smithi002.stdout: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/iscsi-gateway.cfg 2024-09-17T00:23:55.235 INFO:journalctl@ceph.mon.a.smithi002.stdout: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/tcmu-runner-entrypoint.sh 2024-09-17T00:23:55.235 INFO:journalctl@ceph.mon.a.smithi002.stdout: Non-zero exit code 1 from systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem 2024-09-17T00:23:55.235 INFO:journalctl@ceph.mon.a.smithi002.stdout: systemctl: stderr Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:23:55.235 INFO:journalctl@ceph.mon.a.smithi002.stdout: systemctl: stderr See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:23:55.235 INFO:journalctl@ceph.mon.a.smithi002.stdout: Traceback (most recent call last): 2024-09-17T00:23:55.235 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-17T00:23:55.235 INFO:journalctl@ceph.mon.a.smithi002.stdout: return _run_code(code, main_globals, None, 2024-09-17T00:23:55.235 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-17T00:23:55.235 INFO:journalctl@ceph.mon.a.smithi002.stdout: exec(code, run_globals) 2024-09-17T00:23:55.235 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5579, in 2024-09-17T00:23:55.235 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5567, in main 2024-09-17T00:23:55.235 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3049, in command_deploy_from 2024-09-17T00:23:55.235 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3084, in _common_deploy 2024-09-17T00:23:55.235 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3104, in _deploy_daemon_container 2024-09-17T00:23:55.236 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 1124, in deploy_daemon 2024-09-17T00:23:55.236 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-17T00:23:55.236 INFO:journalctl@ceph.mon.a.smithi002.stdout: RuntimeError: Failed command: systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem: Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:23:55.236 INFO:journalctl@ceph.mon.a.smithi002.stdout: See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:23:55.236 INFO:journalctl@ceph.mon.a.smithi002.stdout: Traceback (most recent call last): 2024-09-17T00:23:55.236 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1134, in _check_daemons 2024-09-17T00:23:55.236 INFO:journalctl@ceph.mon.a.smithi002.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-09-17T00:23:55.236 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2473, in _daemon_action 2024-09-17T00:23:55.236 INFO:journalctl@ceph.mon.a.smithi002.stdout: return self.wait_async( 2024-09-17T00:23:55.236 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 796, in wait_async 2024-09-17T00:23:55.236 INFO:journalctl@ceph.mon.a.smithi002.stdout: return self.event_loop.get_result(coro, timeout) 2024-09-17T00:23:55.236 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-09-17T00:23:55.236 INFO:journalctl@ceph.mon.a.smithi002.stdout: return future.result(timeout) 2024-09-17T00:23:55.236 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/lib64/python3.9/concurrent/futures/_base.py", line 446, in result 2024-09-17T00:23:55.237 INFO:journalctl@ceph.mon.a.smithi002.stdout: return self.__get_result() 2024-09-17T00:23:55.237 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/lib64/python3.9/concurrent/futures/_base.py", line 391, in __get_result 2024-09-17T00:23:55.237 INFO:journalctl@ceph.mon.a.smithi002.stdout: raise self._exception 2024-09-17T00:23:55.237 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1370, in _create_daemon 2024-09-17T00:23:55.237 INFO:journalctl@ceph.mon.a.smithi002.stdout: out, err, code = await self._run_cephadm( 2024-09-17T00:23:55.237 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1713, in _run_cephadm 2024-09-17T00:23:55.237 INFO:journalctl@ceph.mon.a.smithi002.stdout: raise OrchestratorError( 2024-09-17T00:23:55.237 INFO:journalctl@ceph.mon.a.smithi002.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi002.vmouem ... 2024-09-17T00:23:55.237 INFO:journalctl@ceph.mon.a.smithi002.stdout: Creating ceph-iscsi config... 2024-09-17T00:23:55.237 INFO:journalctl@ceph.mon.a.smithi002.stdout: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/iscsi-gateway.cfg 2024-09-17T00:23:55.237 INFO:journalctl@ceph.mon.a.smithi002.stdout: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/tcmu-runner-entrypoint.sh 2024-09-17T00:23:55.237 INFO:journalctl@ceph.mon.a.smithi002.stdout: Non-zero exit code 1 from systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem 2024-09-17T00:23:55.237 INFO:journalctl@ceph.mon.a.smithi002.stdout: systemctl: stderr Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:23:55.238 INFO:journalctl@ceph.mon.a.smithi002.stdout: systemctl: stderr See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:23:55.238 INFO:journalctl@ceph.mon.a.smithi002.stdout: Traceback (most recent call last): 2024-09-17T00:23:55.238 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-17T00:23:55.238 INFO:journalctl@ceph.mon.a.smithi002.stdout: return _run_code(code, main_globals, None, 2024-09-17T00:23:55.238 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-17T00:23:55.238 INFO:journalctl@ceph.mon.a.smithi002.stdout: exec(code, run_globals) 2024-09-17T00:23:55.238 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5579, in 2024-09-17T00:23:55.238 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5567, in main 2024-09-17T00:23:55.238 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3049, in command_deploy_from 2024-09-17T00:23:55.238 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3084, in _common_deploy 2024-09-17T00:23:55.238 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3104, in _deploy_daemon_container 2024-09-17T00:23:55.239 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 1124, in deploy_daemon 2024-09-17T00:23:55.239 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-17T00:23:55.239 INFO:journalctl@ceph.mon.a.smithi002.stdout: RuntimeError: Failed command: systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem: Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:23:55.239 INFO:journalctl@ceph.mon.a.smithi002.stdout: See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:23:55.239 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:55 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T00:23:55.239 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:55 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:23:55.239 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:55 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-09-17T00:23:55.239 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:55 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["c"]}]: dispatch 2024-09-17T00:23:55.239 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:55 smithi002 ceph-mon[63804]: Upgrade: It appears safe to stop mon.c 2024-09-17T00:23:55.239 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:55 smithi002 ceph-mon[28203]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T00:23:55.239 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:55 smithi002 ceph-mon[28203]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:23:55.240 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:55 smithi002 ceph-mon[28203]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-09-17T00:23:55.240 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:55 smithi002 ceph-mon[28203]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["c"]}]: dispatch 2024-09-17T00:23:55.240 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:55 smithi002 ceph-mon[28203]: Upgrade: It appears safe to stop mon.c 2024-09-17T00:23:55.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:55 smithi099 ceph-mon[24594]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-17T00:23:55.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:55 smithi099 ceph-mon[24594]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi002.vmouem ... 2024-09-17T00:23:55.467 INFO:journalctl@ceph.mon.b.smithi099.stdout: Creating ceph-iscsi config... 2024-09-17T00:23:55.467 INFO:journalctl@ceph.mon.b.smithi099.stdout: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/iscsi-gateway.cfg 2024-09-17T00:23:55.468 INFO:journalctl@ceph.mon.b.smithi099.stdout: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/tcmu-runner-entrypoint.sh 2024-09-17T00:23:55.468 INFO:journalctl@ceph.mon.b.smithi099.stdout: Non-zero exit code 1 from systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem 2024-09-17T00:23:55.468 INFO:journalctl@ceph.mon.b.smithi099.stdout: systemctl: stderr Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:23:55.468 INFO:journalctl@ceph.mon.b.smithi099.stdout: systemctl: stderr See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:23:55.468 INFO:journalctl@ceph.mon.b.smithi099.stdout: Traceback (most recent call last): 2024-09-17T00:23:55.468 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-17T00:23:55.468 INFO:journalctl@ceph.mon.b.smithi099.stdout: return _run_code(code, main_globals, None, 2024-09-17T00:23:55.468 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-17T00:23:55.468 INFO:journalctl@ceph.mon.b.smithi099.stdout: exec(code, run_globals) 2024-09-17T00:23:55.468 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5579, in 2024-09-17T00:23:55.468 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5567, in main 2024-09-17T00:23:55.468 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3049, in command_deploy_from 2024-09-17T00:23:55.468 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3084, in _common_deploy 2024-09-17T00:23:55.468 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3104, in _deploy_daemon_container 2024-09-17T00:23:55.469 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 1124, in deploy_daemon 2024-09-17T00:23:55.469 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-17T00:23:55.469 INFO:journalctl@ceph.mon.b.smithi099.stdout: RuntimeError: Failed command: systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem: Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:23:55.469 INFO:journalctl@ceph.mon.b.smithi099.stdout: See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:23:55.469 INFO:journalctl@ceph.mon.b.smithi099.stdout: Traceback (most recent call last): 2024-09-17T00:23:55.469 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1134, in _check_daemons 2024-09-17T00:23:55.469 INFO:journalctl@ceph.mon.b.smithi099.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-09-17T00:23:55.469 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2473, in _daemon_action 2024-09-17T00:23:55.469 INFO:journalctl@ceph.mon.b.smithi099.stdout: return self.wait_async( 2024-09-17T00:23:55.469 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 796, in wait_async 2024-09-17T00:23:55.469 INFO:journalctl@ceph.mon.b.smithi099.stdout: return self.event_loop.get_result(coro, timeout) 2024-09-17T00:23:55.469 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-09-17T00:23:55.469 INFO:journalctl@ceph.mon.b.smithi099.stdout: return future.result(timeout) 2024-09-17T00:23:55.469 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/lib64/python3.9/concurrent/futures/_base.py", line 446, in result 2024-09-17T00:23:55.469 INFO:journalctl@ceph.mon.b.smithi099.stdout: return self.__get_result() 2024-09-17T00:23:55.470 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/lib64/python3.9/concurrent/futures/_base.py", line 391, in __get_result 2024-09-17T00:23:55.470 INFO:journalctl@ceph.mon.b.smithi099.stdout: raise self._exception 2024-09-17T00:23:55.470 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1370, in _create_daemon 2024-09-17T00:23:55.470 INFO:journalctl@ceph.mon.b.smithi099.stdout: out, err, code = await self._run_cephadm( 2024-09-17T00:23:55.470 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1713, in _run_cephadm 2024-09-17T00:23:55.470 INFO:journalctl@ceph.mon.b.smithi099.stdout: raise OrchestratorError( 2024-09-17T00:23:55.470 INFO:journalctl@ceph.mon.b.smithi099.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi002.vmouem ... 2024-09-17T00:23:55.470 INFO:journalctl@ceph.mon.b.smithi099.stdout: Creating ceph-iscsi config... 2024-09-17T00:23:55.470 INFO:journalctl@ceph.mon.b.smithi099.stdout: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/iscsi-gateway.cfg 2024-09-17T00:23:55.470 INFO:journalctl@ceph.mon.b.smithi099.stdout: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/tcmu-runner-entrypoint.sh 2024-09-17T00:23:55.470 INFO:journalctl@ceph.mon.b.smithi099.stdout: Non-zero exit code 1 from systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem 2024-09-17T00:23:55.470 INFO:journalctl@ceph.mon.b.smithi099.stdout: systemctl: stderr Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:23:55.470 INFO:journalctl@ceph.mon.b.smithi099.stdout: systemctl: stderr See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:23:55.470 INFO:journalctl@ceph.mon.b.smithi099.stdout: Traceback (most recent call last): 2024-09-17T00:23:55.471 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-17T00:23:55.471 INFO:journalctl@ceph.mon.b.smithi099.stdout: return _run_code(code, main_globals, None, 2024-09-17T00:23:55.471 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-17T00:23:55.471 INFO:journalctl@ceph.mon.b.smithi099.stdout: exec(code, run_globals) 2024-09-17T00:23:55.471 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5579, in 2024-09-17T00:23:55.471 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5567, in main 2024-09-17T00:23:55.471 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3049, in command_deploy_from 2024-09-17T00:23:55.471 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3084, in _common_deploy 2024-09-17T00:23:55.471 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3104, in _deploy_daemon_container 2024-09-17T00:23:55.471 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 1124, in deploy_daemon 2024-09-17T00:23:55.471 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-17T00:23:55.471 INFO:journalctl@ceph.mon.b.smithi099.stdout: RuntimeError: Failed command: systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem: Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:23:55.471 INFO:journalctl@ceph.mon.b.smithi099.stdout: See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:23:55.472 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:55 smithi099 ceph-mon[24594]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T00:23:55.472 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:55 smithi099 ceph-mon[24594]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:23:55.472 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:55 smithi099 ceph-mon[24594]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-09-17T00:23:55.472 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:55 smithi099 ceph-mon[24594]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["c"]}]: dispatch 2024-09-17T00:23:55.472 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:55 smithi099 ceph-mon[24594]: Upgrade: It appears safe to stop mon.c 2024-09-17T00:23:56.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:56 smithi099 ceph-mon[24594]: Upgrade: Updating mon.c 2024-09-17T00:23:56.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:56 smithi099 ceph-mon[24594]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:23:56.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:56 smithi099 ceph-mon[24594]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-17T00:23:56.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:56 smithi099 ceph-mon[24594]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-17T00:23:56.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:56 smithi099 ceph-mon[24594]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:23:56.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:56 smithi099 ceph-mon[24594]: Deploying daemon mon.c on smithi002 2024-09-17T00:23:56.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:23:56 smithi099 ceph-mon[24594]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-17T00:23:56.978 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:56 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: ::ffff:172.21.15.99 - - [17/Sep/2024:00:23:56] "GET /metrics HTTP/1.1" 200 35360 "" "Prometheus/2.43.0" 2024-09-17T00:23:56.978 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:56 smithi002 ceph-mon[28203]: Upgrade: Updating mon.c 2024-09-17T00:23:56.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:56 smithi002 ceph-mon[28203]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:23:56.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:56 smithi002 ceph-mon[28203]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-17T00:23:56.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:56 smithi002 ceph-mon[28203]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-17T00:23:56.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:56 smithi002 ceph-mon[28203]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:23:56.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:56 smithi002 ceph-mon[28203]: Deploying daemon mon.c on smithi002 2024-09-17T00:23:56.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:56 smithi002 ceph-mon[28203]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-17T00:23:56.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:56 smithi002 ceph-mon[63804]: Upgrade: Updating mon.c 2024-09-17T00:23:56.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:56 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:23:56.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:56 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-17T00:23:56.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:56 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-17T00:23:56.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:56 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:23:56.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:56 smithi002 ceph-mon[63804]: Deploying daemon mon.c on smithi002 2024-09-17T00:23:56.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:23:56 smithi002 ceph-mon[63804]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-17T00:23:57.529 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:57 smithi002 systemd[1]: Stopping Ceph mon.c for 3aad27a2-7489-11ef-bceb-c7b262605968... 2024-09-17T00:23:57.529 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:57 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mon-c[28199]: 2024-09-17T00:23:57.519+0000 7fe4e8871700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-17T00:23:57.529 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:57 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mon-c[28199]: 2024-09-17T00:23:57.519+0000 7fe4e8871700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-09-17T00:23:57.883 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:57 smithi002 podman[66558]: 2024-09-17 00:23:57.581355043 +0000 UTC m=+0.161687628 container died 77b3d2e1489987946f3f9bd00f85f233eab9bb2c9986788543474445a5661df7 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mon-c, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., distribution-scope=public, GIT_CLEAN=True, vcs-type=git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, release=754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.component=centos-stream-container, CEPH_POINT_RELEASE=-17.2.0, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.expose-services=, version=8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.tags=base centos centos-stream, vendor=Red Hat, Inc., architecture=x86_64, io.buildah.version=1.19.8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.display-name=CentOS Stream 8, build-date=2022-05-03T08:36:31.336870, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, name=centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image) 2024-09-17T00:23:57.883 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:57 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: [17/Sep/2024:00:23:57] ENGINE Bus STOPPING 2024-09-17T00:23:57.883 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:57 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: [17/Sep/2024:00:23:57] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-17T00:23:57.884 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:57 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: [17/Sep/2024:00:23:57] ENGINE Bus STOPPED 2024-09-17T00:23:57.884 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:57 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: [17/Sep/2024:00:23:57] ENGINE Bus STARTING 2024-09-17T00:23:57.884 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:57 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: [17/Sep/2024:00:23:57] ENGINE Serving on http://:::9283 2024-09-17T00:23:57.884 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:23:57 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: [17/Sep/2024:00:23:57] ENGINE Bus STARTED 2024-09-17T00:23:58.225 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:57 smithi002 podman[66558]: 2024-09-17 00:23:57.994177686 +0000 UTC m=+0.574510272 container cleanup 77b3d2e1489987946f3f9bd00f85f233eab9bb2c9986788543474445a5661df7 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mon-c, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, architecture=x86_64, version=8, GIT_CLEAN=True, name=centos-stream, release=754, GIT_BRANCH=HEAD, build-date=2022-05-03T08:36:31.336870, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vendor=Red Hat, Inc., com.redhat.component=centos-stream-container, io.openshift.tags=base centos centos-stream, CEPH_POINT_RELEASE=-17.2.0, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.openshift.expose-services=, distribution-scope=public, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, RELEASE=HEAD, io.buildah.version=1.19.8, ceph=True, vcs-type=git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=CentOS Stream 8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb) 2024-09-17T00:23:58.225 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:57 smithi002 bash[66558]: ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mon-c 2024-09-17T00:23:58.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:58 smithi002 podman[66573]: 2024-09-17 00:23:58.228817748 +0000 UTC m=+0.640653818 container remove 77b3d2e1489987946f3f9bd00f85f233eab9bb2c9986788543474445a5661df7 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mon-c, GIT_BRANCH=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.expose-services=, io.openshift.tags=base centos centos-stream, io.buildah.version=1.19.8, io.k8s.display-name=CentOS Stream 8, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.component=centos-stream-container, GIT_CLEAN=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, RELEASE=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-type=git, version=8, name=centos-stream, architecture=x86_64, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, maintainer=Guillaume Abrioux , build-date=2022-05-03T08:36:31.336870, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vendor=Red Hat, Inc., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, release=754, CEPH_POINT_RELEASE=-17.2.0, ceph=True) 2024-09-17T00:23:58.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:58 smithi002 systemd[1]: ceph-3aad27a2-7489-11ef-bceb-c7b262605968@mon.c.service: Deactivated successfully. 2024-09-17T00:23:58.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:58 smithi002 systemd[1]: Stopped Ceph mon.c for 3aad27a2-7489-11ef-bceb-c7b262605968. 2024-09-17T00:23:58.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:58 smithi002 systemd[1]: ceph-3aad27a2-7489-11ef-bceb-c7b262605968@mon.c.service: Consumed 8.483s CPU time. 2024-09-17T00:23:58.967 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:23:58 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[52757]: ts=2024-09-17T00:23:58.659Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd name=CephOSDFlapping index=13 msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.2\", device_class=\"ssd\", hostname=\"smithi002\", instance=\"172.21.15.99:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.2\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"3aad27a2-7489-11ef-bceb-c7b262605968\", cluster_addr=\"172.21.15.2\", device_class=\"ssd\", hostname=\"smithi002\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.2\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-17T00:23:58.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:58 smithi002 systemd[1]: Starting Ceph mon.c for 3aad27a2-7489-11ef-bceb-c7b262605968... 2024-09-17T00:23:58.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:58 smithi002 podman[66713]: 2024-09-17 00:23:58.887746337 +0000 UTC m=+0.114027212 container create 8a510255ae66ca7c1a4d698b4e72ce9f3d0173536a2369511f9fd9c47ba44b7c (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mon-c, RELEASE=squid-c2ddf97, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_CLEAN=True) 2024-09-17T00:23:58.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:58 smithi002 podman[66713]: 2024-09-17 00:23:58.790827031 +0000 UTC m=+0.017107918 image pull ebb0615149def4a3179c82c0cc47af75808300abb9f1142788e5e605a7ac0e5b quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f 2024-09-17T00:23:59.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 podman[66713]: 2024-09-17 00:23:59.107017705 +0000 UTC m=+0.333298588 container init 8a510255ae66ca7c1a4d698b4e72ce9f3d0173536a2369511f9fd9c47ba44b7c (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mon-c, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, ceph=True, GIT_CLEAN=True, RELEASE=squid-c2ddf97, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, CEPH_POINT_RELEASE=) 2024-09-17T00:23:59.480 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 podman[66713]: 2024-09-17 00:23:59.110549403 +0000 UTC m=+0.336830278 container start 8a510255ae66ca7c1a4d698b4e72ce9f3d0173536a2369511f9fd9c47ba44b7c (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mon-c, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=squid-c2ddf97, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.build-date=20240731) 2024-09-17T00:23:59.480 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: set uid:gid to 167:167 (ceph:ceph) 2024-09-17T00:23:59.480 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: ceph version 19.1.1-330-gc2ddf972 (c2ddf9721ef254645201cdbabd9c8f49c30a586f) squid (stable), process ceph-mon, pid 2 2024-09-17T00:23:59.480 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: pidfile_write: ignore empty --pid-file 2024-09-17T00:23:59.480 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: load: jerasure load: lrc 2024-09-17T00:23:59.480 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: RocksDB version: 7.9.2 2024-09-17T00:23:59.480 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Git sha 0 2024-09-17T00:23:59.480 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Compile date 2024-09-16 15:09:42 2024-09-17T00:23:59.480 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: DB SUMMARY 2024-09-17T00:23:59.480 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: DB Session ID: 1CYLJEBL7E95WANLTUBP 2024-09-17T00:23:59.480 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: CURRENT file: CURRENT 2024-09-17T00:23:59.480 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: IDENTITY file: IDENTITY 2024-09-17T00:23:59.481 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: MANIFEST file: MANIFEST-000009 size: 1084 Bytes 2024-09-17T00:23:59.481 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 1, files: 000027.sst 2024-09-17T00:23:59.481 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000025.log size: 491994 ; 2024-09-17T00:23:59.481 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.error_if_exists: 0 2024-09-17T00:23:59.481 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.create_if_missing: 0 2024-09-17T00:23:59.481 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.paranoid_checks: 1 2024-09-17T00:23:59.481 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.flush_verify_memtable_count: 1 2024-09-17T00:23:59.481 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-17T00:23:59.481 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-09-17T00:23:59.481 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.env: 0x557bcda76940 2024-09-17T00:23:59.481 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.fs: PosixFileSystem 2024-09-17T00:23:59.481 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.info_log: 0x557bd00696e0 2024-09-17T00:23:59.481 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.max_file_opening_threads: 16 2024-09-17T00:23:59.482 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.statistics: (nil) 2024-09-17T00:23:59.482 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.use_fsync: 0 2024-09-17T00:23:59.482 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.max_log_file_size: 0 2024-09-17T00:23:59.483 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-17T00:23:59.483 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-17T00:23:59.483 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.keep_log_file_num: 1000 2024-09-17T00:23:59.483 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.recycle_log_file_num: 0 2024-09-17T00:23:59.483 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.allow_fallocate: 1 2024-09-17T00:23:59.483 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.allow_mmap_reads: 0 2024-09-17T00:23:59.483 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.allow_mmap_writes: 0 2024-09-17T00:23:59.483 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.use_direct_reads: 0 2024-09-17T00:23:59.483 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-17T00:23:59.483 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.create_missing_column_families: 0 2024-09-17T00:23:59.483 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.db_log_dir: 2024-09-17T00:23:59.483 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.wal_dir: 2024-09-17T00:23:59.483 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-17T00:23:59.484 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-17T00:23:59.484 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-17T00:23:59.484 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-17T00:23:59.484 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-17T00:23:59.484 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-17T00:23:59.484 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.advise_random_on_open: 1 2024-09-17T00:23:59.484 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.db_write_buffer_size: 0 2024-09-17T00:23:59.484 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.write_buffer_manager: 0x557bd0078dc0 2024-09-17T00:23:59.484 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-17T00:23:59.484 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-17T00:23:59.484 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-17T00:23:59.484 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.rate_limiter: (nil) 2024-09-17T00:23:59.484 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-17T00:23:59.485 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.wal_recovery_mode: 2 2024-09-17T00:23:59.485 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.enable_thread_tracking: 0 2024-09-17T00:23:59.485 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.enable_pipelined_write: 0 2024-09-17T00:23:59.486 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.unordered_write: 0 2024-09-17T00:23:59.486 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-17T00:23:59.486 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-17T00:23:59.486 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-17T00:23:59.486 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-17T00:23:59.486 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.row_cache: None 2024-09-17T00:23:59.486 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.wal_filter: None 2024-09-17T00:23:59.486 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-17T00:23:59.486 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.allow_ingest_behind: 0 2024-09-17T00:23:59.486 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.two_write_queues: 0 2024-09-17T00:23:59.486 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.manual_wal_flush: 0 2024-09-17T00:23:59.486 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.wal_compression: 0 2024-09-17T00:23:59.486 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.atomic_flush: 0 2024-09-17T00:23:59.486 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-17T00:23:59.487 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-17T00:23:59.487 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-17T00:23:59.487 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.log_readahead_size: 0 2024-09-17T00:23:59.487 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-17T00:23:59.487 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.best_efforts_recovery: 0 2024-09-17T00:23:59.487 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-17T00:23:59.487 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-17T00:23:59.487 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.allow_data_in_errors: 0 2024-09-17T00:23:59.487 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.db_host_id: __hostname__ 2024-09-17T00:23:59.487 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.enforce_single_del_contracts: true 2024-09-17T00:23:59.487 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.max_background_jobs: 2 2024-09-17T00:23:59.487 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.max_background_compactions: -1 2024-09-17T00:23:59.487 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.max_subcompactions: 1 2024-09-17T00:23:59.487 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-17T00:23:59.488 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-17T00:23:59.488 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-17T00:23:59.488 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.max_total_wal_size: 0 2024-09-17T00:23:59.488 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-17T00:23:59.488 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-17T00:23:59.488 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-17T00:23:59.488 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-17T00:23:59.488 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.max_open_files: -1 2024-09-17T00:23:59.488 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.bytes_per_sync: 0 2024-09-17T00:23:59.488 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-17T00:23:59.489 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-17T00:23:59.489 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.compaction_readahead_size: 0 2024-09-17T00:23:59.489 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.max_background_flushes: -1 2024-09-17T00:23:59.489 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Compression algorithms supported: 2024-09-17T00:23:59.489 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: kZSTD supported: 0 2024-09-17T00:23:59.489 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: kXpressCompression supported: 0 2024-09-17T00:23:59.489 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: kBZip2Compression supported: 0 2024-09-17T00:23:59.489 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-17T00:23:59.489 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: kLZ4Compression supported: 1 2024-09-17T00:23:59.489 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: kZlibCompression supported: 1 2024-09-17T00:23:59.489 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: kLZ4HCCompression supported: 1 2024-09-17T00:23:59.489 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: kSnappyCompression supported: 1 2024-09-17T00:23:59.489 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-17T00:23:59.490 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: DMutex implementation: pthread_mutex_t 2024-09-17T00:23:59.490 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000009 2024-09-17T00:23:59.490 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-09-17T00:23:59.490 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-17T00:23:59.490 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.merge_operator: 2024-09-17T00:23:59.490 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.compaction_filter: None 2024-09-17T00:23:59.490 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.compaction_filter_factory: None 2024-09-17T00:23:59.490 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.sst_partitioner_factory: None 2024-09-17T00:23:59.490 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-17T00:23:59.490 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-17T00:23:59.490 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x557bd0069320) 2024-09-17T00:23:59.490 INFO:journalctl@ceph.mon.c.smithi002.stdout: cache_index_and_filter_blocks: 1 2024-09-17T00:23:59.490 INFO:journalctl@ceph.mon.c.smithi002.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-17T00:23:59.491 INFO:journalctl@ceph.mon.c.smithi002.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-17T00:23:59.491 INFO:journalctl@ceph.mon.c.smithi002.stdout: pin_top_level_index_and_filter: 1 2024-09-17T00:23:59.491 INFO:journalctl@ceph.mon.c.smithi002.stdout: index_type: 0 2024-09-17T00:23:59.491 INFO:journalctl@ceph.mon.c.smithi002.stdout: data_block_index_type: 0 2024-09-17T00:23:59.491 INFO:journalctl@ceph.mon.c.smithi002.stdout: index_shortening: 1 2024-09-17T00:23:59.491 INFO:journalctl@ceph.mon.c.smithi002.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-17T00:23:59.491 INFO:journalctl@ceph.mon.c.smithi002.stdout: checksum: 4 2024-09-17T00:23:59.491 INFO:journalctl@ceph.mon.c.smithi002.stdout: no_block_cache: 0 2024-09-17T00:23:59.491 INFO:journalctl@ceph.mon.c.smithi002.stdout: block_cache: 0x557bd005d1f0 2024-09-17T00:23:59.491 INFO:journalctl@ceph.mon.c.smithi002.stdout: block_cache_name: BinnedLRUCache 2024-09-17T00:23:59.491 INFO:journalctl@ceph.mon.c.smithi002.stdout: block_cache_options: 2024-09-17T00:23:59.491 INFO:journalctl@ceph.mon.c.smithi002.stdout: capacity : 536870912 2024-09-17T00:23:59.491 INFO:journalctl@ceph.mon.c.smithi002.stdout: num_shard_bits : 4 2024-09-17T00:23:59.491 INFO:journalctl@ceph.mon.c.smithi002.stdout: strict_capacity_limit : 0 2024-09-17T00:23:59.491 INFO:journalctl@ceph.mon.c.smithi002.stdout: high_pri_pool_ratio: 0.000 2024-09-17T00:23:59.492 INFO:journalctl@ceph.mon.c.smithi002.stdout: block_cache_compressed: (nil) 2024-09-17T00:23:59.492 INFO:journalctl@ceph.mon.c.smithi002.stdout: persistent_cache: (nil) 2024-09-17T00:23:59.492 INFO:journalctl@ceph.mon.c.smithi002.stdout: block_size: 4096 2024-09-17T00:23:59.492 INFO:journalctl@ceph.mon.c.smithi002.stdout: block_size_deviation: 10 2024-09-17T00:23:59.492 INFO:journalctl@ceph.mon.c.smithi002.stdout: block_restart_interval: 16 2024-09-17T00:23:59.492 INFO:journalctl@ceph.mon.c.smithi002.stdout: index_block_restart_interval: 1 2024-09-17T00:23:59.492 INFO:journalctl@ceph.mon.c.smithi002.stdout: metadata_block_size: 4096 2024-09-17T00:23:59.492 INFO:journalctl@ceph.mon.c.smithi002.stdout: partition_filters: 0 2024-09-17T00:23:59.492 INFO:journalctl@ceph.mon.c.smithi002.stdout: use_delta_encoding: 1 2024-09-17T00:23:59.492 INFO:journalctl@ceph.mon.c.smithi002.stdout: filter_policy: bloomfilter 2024-09-17T00:23:59.492 INFO:journalctl@ceph.mon.c.smithi002.stdout: whole_key_filtering: 1 2024-09-17T00:23:59.492 INFO:journalctl@ceph.mon.c.smithi002.stdout: verify_compression: 0 2024-09-17T00:23:59.492 INFO:journalctl@ceph.mon.c.smithi002.stdout: read_amp_bytes_per_bit: 0 2024-09-17T00:23:59.492 INFO:journalctl@ceph.mon.c.smithi002.stdout: format_version: 5 2024-09-17T00:23:59.493 INFO:journalctl@ceph.mon.c.smithi002.stdout: enable_index_compression: 1 2024-09-17T00:23:59.493 INFO:journalctl@ceph.mon.c.smithi002.stdout: block_align: 0 2024-09-17T00:23:59.493 INFO:journalctl@ceph.mon.c.smithi002.stdout: max_auto_readahead_size: 262144 2024-09-17T00:23:59.493 INFO:journalctl@ceph.mon.c.smithi002.stdout: prepopulate_block_cache: 0 2024-09-17T00:23:59.493 INFO:journalctl@ceph.mon.c.smithi002.stdout: initial_auto_readahead_size: 8192 2024-09-17T00:23:59.493 INFO:journalctl@ceph.mon.c.smithi002.stdout: num_file_reads_for_auto_readahead: 2 2024-09-17T00:23:59.493 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.write_buffer_size: 33554432 2024-09-17T00:23:59.493 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.max_write_buffer_number: 2 2024-09-17T00:23:59.493 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.compression: NoCompression 2024-09-17T00:23:59.493 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.bottommost_compression: Disabled 2024-09-17T00:23:59.493 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.prefix_extractor: nullptr 2024-09-17T00:23:59.493 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-17T00:23:59.493 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.num_levels: 7 2024-09-17T00:23:59.493 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-17T00:23:59.494 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-17T00:23:59.494 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-17T00:23:59.494 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-17T00:23:59.494 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-17T00:23:59.494 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-17T00:23:59.494 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-17T00:23:59.494 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-17T00:23:59.494 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-17T00:23:59.494 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-17T00:23:59.494 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-09-17T00:23:59.494 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-09-17T00:23:59.494 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-17T00:23:59.494 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.compression_opts.level: 32767 2024-09-17T00:23:59.495 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.compression_opts.strategy: 0 2024-09-17T00:23:59.495 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-17T00:23:59.495 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-17T00:23:59.495 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-09-17T00:23:59.495 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-17T00:23:59.495 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.compression_opts.enabled: false 2024-09-17T00:23:59.495 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-09-17T00:23:59.495 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-17T00:23:59.495 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-17T00:23:59.495 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-17T00:23:59.495 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.target_file_size_base: 67108864 2024-09-17T00:23:59.495 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-17T00:23:59.495 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-17T00:23:59.495 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-17T00:23:59.495 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-17T00:23:59.496 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-17T00:23:59.496 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-17T00:23:59.496 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-17T00:23:59.496 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-17T00:23:59.496 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-17T00:23:59.496 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-17T00:23:59.496 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-17T00:23:59.496 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-17T00:23:59.496 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-17T00:23:59.496 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-09-17T00:23:59.496 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.arena_block_size: 1048576 2024-09-17T00:23:59.496 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-17T00:23:59.497 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-17T00:23:59.497 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.disable_auto_compactions: 0 2024-09-17T00:23:59.497 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-17T00:23:59.497 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-17T00:23:59.497 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-17T00:23:59.497 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-17T00:23:59.497 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-17T00:23:59.497 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-17T00:23:59.497 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-17T00:23:59.497 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-17T00:23:59.497 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-17T00:23:59.497 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-17T00:23:59.497 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-17T00:23:59.498 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.inplace_update_support: 0 2024-09-17T00:23:59.498 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-17T00:23:59.498 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-17T00:23:59.498 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-17T00:23:59.498 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-17T00:23:59.498 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.bloom_locality: 0 2024-09-17T00:23:59.498 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.max_successive_merges: 0 2024-09-17T00:23:59.498 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-17T00:23:59.498 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.paranoid_file_checks: 0 2024-09-17T00:23:59.498 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.force_consistency_checks: 1 2024-09-17T00:23:59.498 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.report_bg_io_stats: 0 2024-09-17T00:23:59.498 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.ttl: 2592000 2024-09-17T00:23:59.498 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-17T00:23:59.498 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-09-17T00:23:59.499 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-09-17T00:23:59.499 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.enable_blob_files: false 2024-09-17T00:23:59.499 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.min_blob_size: 0 2024-09-17T00:23:59.499 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.blob_file_size: 268435456 2024-09-17T00:23:59.500 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-17T00:23:59.500 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-17T00:23:59.500 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-17T00:23:59.500 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-09-17T00:23:59.500 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-09-17T00:23:59.500 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.blob_file_starting_level: 0 2024-09-17T00:23:59.500 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-09-17T00:23:59.500 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 27.sst 2024-09-17T00:23:59.500 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-09-17T00:23:59.500 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 29, last_sequence is 15128, log_number is 25,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-09-17T00:23:59.500 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 25 2024-09-17T00:23:59.500 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 0ce868a4-e1c2-49ed-a167-f4ec8e7d5d34 2024-09-17T00:23:59.500 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726532639137875, "job": 1, "event": "recovery_started", "wal_files": [25]} 2024-09-17T00:23:59.501 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #25 mode 2 2024-09-17T00:23:59.501 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726532639139952, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 30, "file_size": 313652, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 15133, "largest_seqno": 15313, "table_properties": {"data_size": 311679, "index_size": 679, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 261, "raw_key_size": 2445, "raw_average_key_size": 25, "raw_value_size": 309442, "raw_average_value_size": 3257, "num_data_blocks": 28, "num_entries": 95, "num_filter_entries": 95, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1726532639, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "0ce868a4-e1c2-49ed-a167-f4ec8e7d5d34", "db_session_id": "1CYLJEBL7E95WANLTUBP", "orig_file_number": 30, "seqno_to_time_mapping": "N/A"}} 2024-09-17T00:23:59.501 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726532639140053, "job": 1, "event": "recovery_finished"} 2024-09-17T00:23:59.501 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: [db/version_set.cc:5047] Creating manifest 32 2024-09-17T00:23:59.501 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-09-17T00:23:59.501 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000025.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-17T00:23:59.501 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x557bd008ae00 2024-09-17T00:23:59.501 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: DB pointer 0x557bd0186000 2024-09-17T00:23:59.501 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-09-17T00:23:59.501 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-09-17T00:23:59.501 INFO:journalctl@ceph.mon.c.smithi002.stdout: ** DB Stats ** 2024-09-17T00:23:59.501 INFO:journalctl@ceph.mon.c.smithi002.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-17T00:23:59.501 INFO:journalctl@ceph.mon.c.smithi002.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-17T00:23:59.502 INFO:journalctl@ceph.mon.c.smithi002.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-17T00:23:59.502 INFO:journalctl@ceph.mon.c.smithi002.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-17T00:23:59.502 INFO:journalctl@ceph.mon.c.smithi002.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-17T00:23:59.502 INFO:journalctl@ceph.mon.c.smithi002.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-17T00:23:59.502 INFO:journalctl@ceph.mon.c.smithi002.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-17T00:23:59.502 INFO:journalctl@ceph.mon.c.smithi002.stdout: 2024-09-17T00:23:59.502 INFO:journalctl@ceph.mon.c.smithi002.stdout: ** Compaction Stats [default] ** 2024-09-17T00:23:59.502 INFO:journalctl@ceph.mon.c.smithi002.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-17T00:23:59.502 INFO:journalctl@ceph.mon.c.smithi002.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-09-17T00:23:59.502 INFO:journalctl@ceph.mon.c.smithi002.stdout: L0 1/0 306.30 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 238.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-17T00:23:59.502 INFO:journalctl@ceph.mon.c.smithi002.stdout: L6 1/0 11.87 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-09-17T00:23:59.502 INFO:journalctl@ceph.mon.c.smithi002.stdout: Sum 2/0 12.17 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 238.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-17T00:23:59.502 INFO:journalctl@ceph.mon.c.smithi002.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 238.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-17T00:23:59.502 INFO:journalctl@ceph.mon.c.smithi002.stdout: 2024-09-17T00:23:59.503 INFO:journalctl@ceph.mon.c.smithi002.stdout: ** Compaction Stats [default] ** 2024-09-17T00:23:59.503 INFO:journalctl@ceph.mon.c.smithi002.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-17T00:23:59.503 INFO:journalctl@ceph.mon.c.smithi002.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-17T00:23:59.503 INFO:journalctl@ceph.mon.c.smithi002.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 238.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-17T00:23:59.503 INFO:journalctl@ceph.mon.c.smithi002.stdout: 2024-09-17T00:23:59.503 INFO:journalctl@ceph.mon.c.smithi002.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-09-17T00:23:59.503 INFO:journalctl@ceph.mon.c.smithi002.stdout: 2024-09-17T00:23:59.503 INFO:journalctl@ceph.mon.c.smithi002.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-17T00:23:59.503 INFO:journalctl@ceph.mon.c.smithi002.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-17T00:23:59.503 INFO:journalctl@ceph.mon.c.smithi002.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-17T00:23:59.503 INFO:journalctl@ceph.mon.c.smithi002.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-17T00:23:59.503 INFO:journalctl@ceph.mon.c.smithi002.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-17T00:23:59.503 INFO:journalctl@ceph.mon.c.smithi002.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-17T00:23:59.503 INFO:journalctl@ceph.mon.c.smithi002.stdout: Cumulative compaction: 0.00 GB write, 68.18 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-17T00:23:59.504 INFO:journalctl@ceph.mon.c.smithi002.stdout: Interval compaction: 0.00 GB write, 68.18 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-17T00:23:59.504 INFO:journalctl@ceph.mon.c.smithi002.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-17T00:23:59.504 INFO:journalctl@ceph.mon.c.smithi002.stdout: Block cache BinnedLRUCache@0x557bd005d1f0#2 capacity: 512.00 MB usage: 1.08 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-09-17T00:23:59.504 INFO:journalctl@ceph.mon.c.smithi002.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.31 KB,5.96046e-05%) IndexBlock(1,0.77 KB,0.000146031%) Misc(1,0.00 KB,0%) 2024-09-17T00:23:59.504 INFO:journalctl@ceph.mon.c.smithi002.stdout: 2024-09-17T00:23:59.504 INFO:journalctl@ceph.mon.c.smithi002.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-17T00:23:59.504 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: starting mon.c rank 1 at public addrs [v2:172.21.15.2:3301/0,v1:172.21.15.2:6790/0] at bind addrs [v2:172.21.15.2:3301/0,v1:172.21.15.2:6790/0] mon_data /var/lib/ceph/mon/ceph-c fsid 3aad27a2-7489-11ef-bceb-c7b262605968 2024-09-17T00:23:59.504 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: mon.c@-1(???) e3 preinit fsid 3aad27a2-7489-11ef-bceb-c7b262605968 2024-09-17T00:23:59.504 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: mon.c@-1(???).mds e1 new map 2024-09-17T00:23:59.504 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: mon.c@-1(???).mds e1 print_map 2024-09-17T00:23:59.504 INFO:journalctl@ceph.mon.c.smithi002.stdout: e1 2024-09-17T00:23:59.504 INFO:journalctl@ceph.mon.c.smithi002.stdout: btime 1970-01-01T00:00:00:000000+0000 2024-09-17T00:23:59.504 INFO:journalctl@ceph.mon.c.smithi002.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-17T00:23:59.505 INFO:journalctl@ceph.mon.c.smithi002.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-09-17T00:23:59.505 INFO:journalctl@ceph.mon.c.smithi002.stdout: legacy client fscid: -1 2024-09-17T00:23:59.505 INFO:journalctl@ceph.mon.c.smithi002.stdout: 2024-09-17T00:23:59.505 INFO:journalctl@ceph.mon.c.smithi002.stdout: No filesystems configured 2024-09-17T00:23:59.505 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: mon.c@-1(???).osd e106 crush map has features 3314933000854323200, adjusting msgr requires 2024-09-17T00:23:59.505 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: mon.c@-1(???).osd e106 crush map has features 432629239337189376, adjusting msgr requires 2024-09-17T00:23:59.505 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: mon.c@-1(???).osd e106 crush map has features 432629239337189376, adjusting msgr requires 2024-09-17T00:23:59.505 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: mon.c@-1(???).osd e106 crush map has features 432629239337189376, adjusting msgr requires 2024-09-17T00:23:59.505 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 ceph-mon[66732]: mon.c@-1(???).paxosservice(auth 1..23) refresh upgraded, format 0 -> 3 2024-09-17T00:23:59.505 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 bash[66713]: 8a510255ae66ca7c1a4d698b4e72ce9f3d0173536a2369511f9fd9c47ba44b7c 2024-09-17T00:23:59.505 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:23:59 smithi002 systemd[1]: Started Ceph mon.c for 3aad27a2-7489-11ef-bceb-c7b262605968. 2024-09-17T00:24:00.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:00 smithi099 ceph-mon[24594]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T00:24:00.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:00 smithi099 ceph-mon[24594]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T00:24:00.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:00 smithi099 ceph-mon[24594]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T00:24:00.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:00 smithi099 ceph-mon[24594]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T00:24:00.968 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:00 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T00:24:00.968 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:00 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T00:24:00.968 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:00 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T00:24:00.969 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:00 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T00:24:00.969 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:00 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T00:24:00.969 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:00 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T00:24:00.969 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:00 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T00:24:00.969 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:00 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T00:24:01.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:01 smithi099 ceph-mon[24594]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-17T00:24:01.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:01 smithi099 ceph-mon[24594]: mon.c calling monitor election 2024-09-17T00:24:01.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:01 smithi099 ceph-mon[24594]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-17T00:24:01.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:01 smithi099 ceph-mon[24594]: mon.a calling monitor election 2024-09-17T00:24:01.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:01 smithi099 ceph-mon[24594]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-09-17T00:24:01.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:01 smithi099 ceph-mon[24594]: monmap epoch 3 2024-09-17T00:24:01.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:01 smithi099 ceph-mon[24594]: fsid 3aad27a2-7489-11ef-bceb-c7b262605968 2024-09-17T00:24:01.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:01 smithi099 ceph-mon[24594]: last_changed 2024-09-17T00:12:24.385107+0000 2024-09-17T00:24:01.968 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:01 smithi099 ceph-mon[24594]: created 2024-09-17T00:10:42.458448+0000 2024-09-17T00:24:01.968 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:01 smithi099 ceph-mon[24594]: min_mon_release 17 (quincy) 2024-09-17T00:24:01.968 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:01 smithi099 ceph-mon[24594]: election_strategy: 1 2024-09-17T00:24:01.968 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:01 smithi099 ceph-mon[24594]: 0: [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] mon.a 2024-09-17T00:24:01.968 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:01 smithi099 ceph-mon[24594]: 1: [v2:172.21.15.2:3301/0,v1:172.21.15.2:6790/0] mon.c 2024-09-17T00:24:01.968 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:01 smithi099 ceph-mon[24594]: 2: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.b 2024-09-17T00:24:01.968 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:01 smithi099 ceph-mon[24594]: fsmap 2024-09-17T00:24:01.968 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:01 smithi099 ceph-mon[24594]: osdmap e106: 8 total, 8 up, 8 in 2024-09-17T00:24:01.968 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:01 smithi099 ceph-mon[24594]: mgrmap e40: y(active, since 18s), standbys: x 2024-09-17T00:24:01.968 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:01 smithi099 ceph-mon[24594]: overall HEALTH_OK 2024-09-17T00:24:01.968 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:01 smithi099 ceph-mon[24594]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:01.968 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:01 smithi099 ceph-mon[24594]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:01.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:01 smithi002 ceph-mon[66732]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-17T00:24:01.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:01 smithi002 ceph-mon[66732]: mon.c calling monitor election 2024-09-17T00:24:01.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:01 smithi002 ceph-mon[66732]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-17T00:24:01.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:01 smithi002 ceph-mon[66732]: mon.a calling monitor election 2024-09-17T00:24:01.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:01 smithi002 ceph-mon[66732]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-09-17T00:24:01.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:01 smithi002 ceph-mon[66732]: monmap epoch 3 2024-09-17T00:24:01.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:01 smithi002 ceph-mon[66732]: fsid 3aad27a2-7489-11ef-bceb-c7b262605968 2024-09-17T00:24:01.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:01 smithi002 ceph-mon[66732]: last_changed 2024-09-17T00:12:24.385107+0000 2024-09-17T00:24:01.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:01 smithi002 ceph-mon[66732]: created 2024-09-17T00:10:42.458448+0000 2024-09-17T00:24:01.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:01 smithi002 ceph-mon[66732]: min_mon_release 17 (quincy) 2024-09-17T00:24:01.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:01 smithi002 ceph-mon[66732]: election_strategy: 1 2024-09-17T00:24:01.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:01 smithi002 ceph-mon[66732]: 0: [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] mon.a 2024-09-17T00:24:01.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:01 smithi002 ceph-mon[66732]: 1: [v2:172.21.15.2:3301/0,v1:172.21.15.2:6790/0] mon.c 2024-09-17T00:24:01.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:01 smithi002 ceph-mon[66732]: 2: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.b 2024-09-17T00:24:01.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:01 smithi002 ceph-mon[66732]: fsmap 2024-09-17T00:24:01.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:01 smithi002 ceph-mon[66732]: osdmap e106: 8 total, 8 up, 8 in 2024-09-17T00:24:01.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:01 smithi002 ceph-mon[66732]: mgrmap e40: y(active, since 18s), standbys: x 2024-09-17T00:24:01.981 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:01 smithi002 ceph-mon[66732]: overall HEALTH_OK 2024-09-17T00:24:01.981 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:01 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:01.981 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:01 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:01.981 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:01 smithi002 ceph-mon[63804]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-17T00:24:01.981 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:01 smithi002 ceph-mon[63804]: mon.c calling monitor election 2024-09-17T00:24:01.981 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:01 smithi002 ceph-mon[63804]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-09-17T00:24:01.981 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:01 smithi002 ceph-mon[63804]: mon.a calling monitor election 2024-09-17T00:24:01.981 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:01 smithi002 ceph-mon[63804]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-09-17T00:24:01.981 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:01 smithi002 ceph-mon[63804]: monmap epoch 3 2024-09-17T00:24:01.981 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:01 smithi002 ceph-mon[63804]: fsid 3aad27a2-7489-11ef-bceb-c7b262605968 2024-09-17T00:24:01.981 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:01 smithi002 ceph-mon[63804]: last_changed 2024-09-17T00:12:24.385107+0000 2024-09-17T00:24:01.982 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:01 smithi002 ceph-mon[63804]: created 2024-09-17T00:10:42.458448+0000 2024-09-17T00:24:01.982 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:01 smithi002 ceph-mon[63804]: min_mon_release 17 (quincy) 2024-09-17T00:24:01.982 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:01 smithi002 ceph-mon[63804]: election_strategy: 1 2024-09-17T00:24:01.982 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:01 smithi002 ceph-mon[63804]: 0: [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] mon.a 2024-09-17T00:24:01.982 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:01 smithi002 ceph-mon[63804]: 1: [v2:172.21.15.2:3301/0,v1:172.21.15.2:6790/0] mon.c 2024-09-17T00:24:01.982 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:01 smithi002 ceph-mon[63804]: 2: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.b 2024-09-17T00:24:01.982 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:01 smithi002 ceph-mon[63804]: fsmap 2024-09-17T00:24:01.982 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:01 smithi002 ceph-mon[63804]: osdmap e106: 8 total, 8 up, 8 in 2024-09-17T00:24:01.982 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:01 smithi002 ceph-mon[63804]: mgrmap e40: y(active, since 18s), standbys: x 2024-09-17T00:24:01.982 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:01 smithi002 ceph-mon[63804]: overall HEALTH_OK 2024-09-17T00:24:01.982 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:01 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:01.982 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:01 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:02.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:02 smithi099 ceph-mon[24594]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-09-17T00:24:02.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:02 smithi002 ceph-mon[66732]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-09-17T00:24:02.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:02 smithi002 ceph-mon[63804]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-09-17T00:24:03.967 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:24:03 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[52757]: ts=2024-09-17T00:24:03.506Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes name=CephNodeDiskspaceWarning index=4 msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi002\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi002\", job=\"node\", machine=\"x86_64\", nodename=\"smithi002\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"3aad27a2-7489-11ef-bceb-c7b262605968\", domainname=\"(none)\", instance=\"smithi002\", job=\"node\", machine=\"x86_64\", nodename=\"smithi002\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-17T00:24:04.467 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:24:04 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: 2024-09-17T00:24:04.150+0000 7f17efec7640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-09-17T00:24:05.324 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:05 smithi002 ceph-mon[66732]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-09-17T00:24:05.324 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:05 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:05.324 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:05 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:05.324 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:05 smithi002 ceph-mon[63804]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-09-17T00:24:05.325 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:05 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:05.325 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:05 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:05.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:05 smithi099 ceph-mon[24594]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-09-17T00:24:05.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:05 smithi099 ceph-mon[24594]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:05.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:05 smithi099 ceph-mon[24594]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:06.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:06 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:06.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:06 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:06.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:06 smithi002 ceph-mon[66732]: from='client.? 172.21.15.2:0/812175650' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-17T00:24:06.480 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:06 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:06.480 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:06 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:06.480 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:06 smithi002 ceph-mon[63804]: from='client.? 172.21.15.2:0/812175650' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-17T00:24:06.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:06 smithi099 ceph-mon[24594]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:06.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:06 smithi099 ceph-mon[24594]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:06.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:06 smithi099 ceph-mon[24594]: from='client.? 172.21.15.2:0/812175650' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-17T00:24:06.978 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:24:06 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: ::ffff:172.21.15.99 - - [17/Sep/2024:00:24:06] "GET /metrics HTTP/1.1" 200 38161 "" "Prometheus/2.43.0" 2024-09-17T00:24:07.228 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:07 smithi002 ceph-mon[63804]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:24:07.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:07 smithi002 ceph-mon[63804]: from='client.? 172.21.15.2:0/588764265' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6800/627921530"}]: dispatch 2024-09-17T00:24:07.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:07 smithi002 ceph-mon[66732]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:24:07.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:07 smithi002 ceph-mon[66732]: from='client.? 172.21.15.2:0/588764265' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6800/627921530"}]: dispatch 2024-09-17T00:24:07.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:07 smithi099 ceph-mon[24594]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:24:07.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:07 smithi099 ceph-mon[24594]: from='client.? 172.21.15.2:0/588764265' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6800/627921530"}]: dispatch 2024-09-17T00:24:08.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:08 smithi002 ceph-mon[66732]: from='client.? 172.21.15.2:0/588764265' entity='client.iscsi.foo.smithi002.vmouem' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6800/627921530"}]': finished 2024-09-17T00:24:08.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:08 smithi002 ceph-mon[66732]: osdmap e107: 8 total, 8 up, 8 in 2024-09-17T00:24:08.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:08 smithi002 ceph-mon[66732]: from='client.? 172.21.15.2:0/2104539789' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/1607296610"}]: dispatch 2024-09-17T00:24:08.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:08 smithi002 ceph-mon[66732]: from='client.? ' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/1607296610"}]: dispatch 2024-09-17T00:24:08.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:08 smithi002 ceph-mon[63804]: from='client.? 172.21.15.2:0/588764265' entity='client.iscsi.foo.smithi002.vmouem' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6800/627921530"}]': finished 2024-09-17T00:24:08.480 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:08 smithi002 ceph-mon[63804]: osdmap e107: 8 total, 8 up, 8 in 2024-09-17T00:24:08.480 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:08 smithi002 ceph-mon[63804]: from='client.? 172.21.15.2:0/2104539789' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/1607296610"}]: dispatch 2024-09-17T00:24:08.480 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:08 smithi002 ceph-mon[63804]: from='client.? ' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/1607296610"}]: dispatch 2024-09-17T00:24:08.656 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:08 smithi099 ceph-mon[24594]: from='client.? 172.21.15.2:0/588764265' entity='client.iscsi.foo.smithi002.vmouem' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6800/627921530"}]': finished 2024-09-17T00:24:08.656 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:08 smithi099 ceph-mon[24594]: osdmap e107: 8 total, 8 up, 8 in 2024-09-17T00:24:08.656 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:08 smithi099 ceph-mon[24594]: from='client.? 172.21.15.2:0/2104539789' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/1607296610"}]: dispatch 2024-09-17T00:24:08.656 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:08 smithi099 ceph-mon[24594]: from='client.? ' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/1607296610"}]: dispatch 2024-09-17T00:24:08.967 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:24:08 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[52757]: ts=2024-09-17T00:24:08.659Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd name=CephOSDFlapping index=13 msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.2\", device_class=\"ssd\", hostname=\"smithi002\", instance=\"172.21.15.99:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.2\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"3aad27a2-7489-11ef-bceb-c7b262605968\", cluster_addr=\"172.21.15.2\", device_class=\"ssd\", hostname=\"smithi002\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.2\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-17T00:24:09.492 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:09 smithi002 ceph-mon[63804]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:24:09.492 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:09 smithi002 ceph-mon[63804]: from='client.? ' entity='client.iscsi.foo.smithi002.vmouem' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/1607296610"}]': finished 2024-09-17T00:24:09.492 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:09 smithi002 ceph-mon[63804]: osdmap e108: 8 total, 8 up, 8 in 2024-09-17T00:24:09.493 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:09 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:09.493 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:09 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:09.493 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:09 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:24:09.493 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:09 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:24:09.493 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:09 smithi002 ceph-mon[63804]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 450 B/s rd, 0 op/s 2024-09-17T00:24:09.493 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:09 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:09.493 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:09 smithi002 ceph-mon[63804]: from='client.? 172.21.15.2:0/1552040116' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/2832543502"}]: dispatch 2024-09-17T00:24:09.493 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:09 smithi002 ceph-mon[63804]: from='client.? ' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/2832543502"}]: dispatch 2024-09-17T00:24:09.493 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:09 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi002.vmouem", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-17T00:24:09.493 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:09 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi002.vmouem", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-17T00:24:09.493 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:09 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:24:09.494 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:09 smithi002 ceph-mon[66732]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:24:09.494 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:09 smithi002 ceph-mon[66732]: from='client.? ' entity='client.iscsi.foo.smithi002.vmouem' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/1607296610"}]': finished 2024-09-17T00:24:09.494 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:09 smithi002 ceph-mon[66732]: osdmap e108: 8 total, 8 up, 8 in 2024-09-17T00:24:09.494 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:09 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:09.494 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:09 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:09.494 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:09 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:24:09.494 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:09 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:24:09.494 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:09 smithi002 ceph-mon[66732]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 450 B/s rd, 0 op/s 2024-09-17T00:24:09.494 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:09 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:09.494 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:09 smithi002 ceph-mon[66732]: from='client.? 172.21.15.2:0/1552040116' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/2832543502"}]: dispatch 2024-09-17T00:24:09.494 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:09 smithi002 ceph-mon[66732]: from='client.? ' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/2832543502"}]: dispatch 2024-09-17T00:24:09.494 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:09 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi002.vmouem", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-17T00:24:09.494 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:09 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi002.vmouem", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-17T00:24:09.495 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:09 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:24:09.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:09 smithi099 ceph-mon[24594]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:24:09.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:09 smithi099 ceph-mon[24594]: from='client.? ' entity='client.iscsi.foo.smithi002.vmouem' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/1607296610"}]': finished 2024-09-17T00:24:09.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:09 smithi099 ceph-mon[24594]: osdmap e108: 8 total, 8 up, 8 in 2024-09-17T00:24:09.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:09 smithi099 ceph-mon[24594]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:09.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:09 smithi099 ceph-mon[24594]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:09.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:09 smithi099 ceph-mon[24594]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:24:09.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:09 smithi099 ceph-mon[24594]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:24:09.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:09 smithi099 ceph-mon[24594]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 450 B/s rd, 0 op/s 2024-09-17T00:24:09.718 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:09 smithi099 ceph-mon[24594]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:09.718 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:09 smithi099 ceph-mon[24594]: from='client.? 172.21.15.2:0/1552040116' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/2832543502"}]: dispatch 2024-09-17T00:24:09.718 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:09 smithi099 ceph-mon[24594]: from='client.? ' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/2832543502"}]: dispatch 2024-09-17T00:24:09.718 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:09 smithi099 ceph-mon[24594]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi002.vmouem", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-17T00:24:09.718 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:09 smithi099 ceph-mon[24594]: from='mgr.25256 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi002.vmouem", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-17T00:24:09.718 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:09 smithi099 ceph-mon[24594]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:24:10.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:10 smithi002 ceph-mon[66732]: Reconfiguring iscsi.foo.smithi002.vmouem (dependencies changed)... 2024-09-17T00:24:10.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:10 smithi002 ceph-mon[66732]: Reconfiguring daemon iscsi.foo.smithi002.vmouem on smithi002 2024-09-17T00:24:10.480 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:10 smithi002 ceph-mon[66732]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-09-17T00:24:10.480 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:10 smithi002 ceph-mon[66732]: from='client.? ' entity='client.iscsi.foo.smithi002.vmouem' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/2832543502"}]': finished 2024-09-17T00:24:10.480 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:10 smithi002 ceph-mon[66732]: osdmap e109: 8 total, 8 up, 8 in 2024-09-17T00:24:10.480 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:10 smithi002 ceph-mon[66732]: from='client.? 172.21.15.2:0/1155524200' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/2945966602"}]: dispatch 2024-09-17T00:24:10.480 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:10 smithi002 ceph-mon[66732]: from='client.? ' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/2945966602"}]: dispatch 2024-09-17T00:24:10.480 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:10 smithi002 ceph-mon[63804]: Reconfiguring iscsi.foo.smithi002.vmouem (dependencies changed)... 2024-09-17T00:24:10.481 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:10 smithi002 ceph-mon[63804]: Reconfiguring daemon iscsi.foo.smithi002.vmouem on smithi002 2024-09-17T00:24:10.481 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:10 smithi002 ceph-mon[63804]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-09-17T00:24:10.481 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:10 smithi002 ceph-mon[63804]: from='client.? ' entity='client.iscsi.foo.smithi002.vmouem' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/2832543502"}]': finished 2024-09-17T00:24:10.481 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:10 smithi002 ceph-mon[63804]: osdmap e109: 8 total, 8 up, 8 in 2024-09-17T00:24:10.481 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:10 smithi002 ceph-mon[63804]: from='client.? 172.21.15.2:0/1155524200' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/2945966602"}]: dispatch 2024-09-17T00:24:10.481 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:10 smithi002 ceph-mon[63804]: from='client.? ' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/2945966602"}]: dispatch 2024-09-17T00:24:10.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:10 smithi099 ceph-mon[24594]: Reconfiguring iscsi.foo.smithi002.vmouem (dependencies changed)... 2024-09-17T00:24:10.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:10 smithi099 ceph-mon[24594]: Reconfiguring daemon iscsi.foo.smithi002.vmouem on smithi002 2024-09-17T00:24:10.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:10 smithi099 ceph-mon[24594]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-09-17T00:24:10.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:10 smithi099 ceph-mon[24594]: from='client.? ' entity='client.iscsi.foo.smithi002.vmouem' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/2832543502"}]': finished 2024-09-17T00:24:10.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:10 smithi099 ceph-mon[24594]: osdmap e109: 8 total, 8 up, 8 in 2024-09-17T00:24:10.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:10 smithi099 ceph-mon[24594]: from='client.? 172.21.15.2:0/1155524200' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/2945966602"}]: dispatch 2024-09-17T00:24:10.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:10 smithi099 ceph-mon[24594]: from='client.? ' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/2945966602"}]: dispatch 2024-09-17T00:24:11.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:11 smithi002 ceph-mon[66732]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 637 B/s rd, 0 op/s 2024-09-17T00:24:11.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:11 smithi002 ceph-mon[66732]: from='client.? ' entity='client.iscsi.foo.smithi002.vmouem' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/2945966602"}]': finished 2024-09-17T00:24:11.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:11 smithi002 ceph-mon[66732]: osdmap e110: 8 total, 8 up, 8 in 2024-09-17T00:24:11.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:11 smithi002 ceph-mon[63804]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 637 B/s rd, 0 op/s 2024-09-17T00:24:11.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:11 smithi002 ceph-mon[63804]: from='client.? ' entity='client.iscsi.foo.smithi002.vmouem' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/2945966602"}]': finished 2024-09-17T00:24:11.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:11 smithi002 ceph-mon[63804]: osdmap e110: 8 total, 8 up, 8 in 2024-09-17T00:24:11.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:11 smithi099 ceph-mon[24594]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 637 B/s rd, 0 op/s 2024-09-17T00:24:11.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:11 smithi099 ceph-mon[24594]: from='client.? ' entity='client.iscsi.foo.smithi002.vmouem' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/2945966602"}]': finished 2024-09-17T00:24:11.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:11 smithi099 ceph-mon[24594]: osdmap e110: 8 total, 8 up, 8 in 2024-09-17T00:24:12.230 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:24:11 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: 2024-09-17T00:24:11.830+0000 7f17e1e2b640 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi002.vmouem ... 2024-09-17T00:24:12.230 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:24:11 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: Creating ceph-iscsi config... 2024-09-17T00:24:12.230 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:24:11 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/iscsi-gateway.cfg 2024-09-17T00:24:12.230 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:24:11 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/tcmu-runner-entrypoint.sh 2024-09-17T00:24:12.230 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:24:11 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: Non-zero exit code 1 from systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem 2024-09-17T00:24:12.230 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:24:11 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: systemctl: stderr Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:24:12.230 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:24:11 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: systemctl: stderr See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:24:12.230 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:24:11 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: Traceback (most recent call last): 2024-09-17T00:24:12.230 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:24:11 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-17T00:24:12.231 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:24:11 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: return _run_code(code, main_globals, None, 2024-09-17T00:24:12.231 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:24:11 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-17T00:24:12.231 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:24:11 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: exec(code, run_globals) 2024-09-17T00:24:12.231 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:24:11 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5579, in 2024-09-17T00:24:12.231 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:24:11 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5567, in main 2024-09-17T00:24:12.231 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:24:11 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3049, in command_deploy_from 2024-09-17T00:24:12.231 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:24:11 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3084, in _common_deploy 2024-09-17T00:24:12.231 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:24:11 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3104, in _deploy_daemon_container 2024-09-17T00:24:12.231 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:24:11 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 1124, in deploy_daemon 2024-09-17T00:24:12.231 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:24:11 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/tmp/tmpnroy8d0w.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-17T00:24:12.231 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:24:11 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: RuntimeError: Failed command: systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem: Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:24:12.231 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:24:11 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:24:12.232 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:24:11 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: Traceback (most recent call last): 2024-09-17T00:24:12.232 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:24:11 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1134, in _check_daemons 2024-09-17T00:24:12.232 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:24:11 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: self.mgr._daemon_action(daemon_spec, action=action) 2024-09-17T00:24:12.232 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:24:11 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2473, in _daemon_action 2024-09-17T00:24:12.232 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:24:11 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: return self.wait_async( 2024-09-17T00:24:12.232 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:24:11 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/usr/share/ceph/mgr/cephadm/module.py", line 796, in wait_async 2024-09-17T00:24:12.232 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:24:11 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: return self.event_loop.get_result(coro, timeout) 2024-09-17T00:24:12.232 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:24:11 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-09-17T00:24:12.232 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:24:11 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: return future.result(timeout) 2024-09-17T00:24:12.233 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:24:11 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/lib64/python3.9/concurrent/futures/_base.py", line 446, in result 2024-09-17T00:24:12.233 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:24:11 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: return self.__get_result() 2024-09-17T00:24:12.233 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:24:11 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/lib64/python3.9/concurrent/futures/_base.py", line 391, in __get_result 2024-09-17T00:24:12.233 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:24:11 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: raise self._exception 2024-09-17T00:24:12.233 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:24:11 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1370, in _create_daemon 2024-09-17T00:24:12.233 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:24:11 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: out, err, code = await self._run_cephadm( 2024-09-17T00:24:12.233 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:24:11 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1713, in _run_cephadm 2024-09-17T00:24:12.233 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:24:11 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: raise OrchestratorError( 2024-09-17T00:24:12.233 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:24:11 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi002.vmouem ... 2024-09-17T00:24:12.233 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:24:11 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: Creating ceph-iscsi config... 2024-09-17T00:24:12.233 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:24:11 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/iscsi-gateway.cfg 2024-09-17T00:24:12.233 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:24:11 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/tcmu-runner-entrypoint.sh 2024-09-17T00:24:12.233 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:24:11 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: Non-zero exit code 1 from systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem 2024-09-17T00:24:12.233 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:24:11 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: systemctl: stderr Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:24:12.234 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:24:11 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: systemctl: stderr See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:24:12.234 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:24:11 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: Traceback (most recent call last): 2024-09-17T00:24:12.234 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:24:11 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-17T00:24:12.234 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:24:11 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: return _run_code(code, main_globals, None, 2024-09-17T00:24:12.234 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:24:11 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-17T00:24:12.234 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:24:11 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: exec(code, run_globals) 2024-09-17T00:24:12.234 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:24:11 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5579, in 2024-09-17T00:24:12.234 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:24:11 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5567, in main 2024-09-17T00:24:12.234 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:24:11 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3049, in command_deploy_from 2024-09-17T00:24:12.234 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:24:11 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3084, in _common_deploy 2024-09-17T00:24:12.234 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:24:11 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3104, in _deploy_daemon_container 2024-09-17T00:24:12.234 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:24:11 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 1124, in deploy_daemon 2024-09-17T00:24:12.234 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:24:11 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/tmp/tmpnroy8d0w.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-17T00:24:12.234 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:24:11 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: RuntimeError: Failed command: systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem: Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:24:12.235 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:24:11 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:24:13.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:12 smithi099 ceph-mon[24594]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:13.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:12 smithi099 ceph-mon[24594]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T00:24:13.218 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:12 smithi099 ceph-mon[24594]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi002.vmouem ... 2024-09-17T00:24:13.218 INFO:journalctl@ceph.mon.b.smithi099.stdout: Creating ceph-iscsi config... 2024-09-17T00:24:13.218 INFO:journalctl@ceph.mon.b.smithi099.stdout: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/iscsi-gateway.cfg 2024-09-17T00:24:13.218 INFO:journalctl@ceph.mon.b.smithi099.stdout: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/tcmu-runner-entrypoint.sh 2024-09-17T00:24:13.218 INFO:journalctl@ceph.mon.b.smithi099.stdout: Non-zero exit code 1 from systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem 2024-09-17T00:24:13.218 INFO:journalctl@ceph.mon.b.smithi099.stdout: systemctl: stderr Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:24:13.218 INFO:journalctl@ceph.mon.b.smithi099.stdout: systemctl: stderr See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:24:13.218 INFO:journalctl@ceph.mon.b.smithi099.stdout: Traceback (most recent call last): 2024-09-17T00:24:13.218 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-17T00:24:13.218 INFO:journalctl@ceph.mon.b.smithi099.stdout: return _run_code(code, main_globals, None, 2024-09-17T00:24:13.218 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-17T00:24:13.218 INFO:journalctl@ceph.mon.b.smithi099.stdout: exec(code, run_globals) 2024-09-17T00:24:13.218 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5579, in 2024-09-17T00:24:13.218 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5567, in main 2024-09-17T00:24:13.219 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3049, in command_deploy_from 2024-09-17T00:24:13.219 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3084, in _common_deploy 2024-09-17T00:24:13.219 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3104, in _deploy_daemon_container 2024-09-17T00:24:13.219 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 1124, in deploy_daemon 2024-09-17T00:24:13.219 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-17T00:24:13.219 INFO:journalctl@ceph.mon.b.smithi099.stdout: RuntimeError: Failed command: systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem: Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:24:13.219 INFO:journalctl@ceph.mon.b.smithi099.stdout: See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:24:13.219 INFO:journalctl@ceph.mon.b.smithi099.stdout: Traceback (most recent call last): 2024-09-17T00:24:13.219 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1134, in _check_daemons 2024-09-17T00:24:13.219 INFO:journalctl@ceph.mon.b.smithi099.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-09-17T00:24:13.219 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2473, in _daemon_action 2024-09-17T00:24:13.219 INFO:journalctl@ceph.mon.b.smithi099.stdout: return self.wait_async( 2024-09-17T00:24:13.219 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 796, in wait_async 2024-09-17T00:24:13.220 INFO:journalctl@ceph.mon.b.smithi099.stdout: return self.event_loop.get_result(coro, timeout) 2024-09-17T00:24:13.220 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-09-17T00:24:13.220 INFO:journalctl@ceph.mon.b.smithi099.stdout: return future.result(timeout) 2024-09-17T00:24:13.220 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/lib64/python3.9/concurrent/futures/_base.py", line 446, in result 2024-09-17T00:24:13.220 INFO:journalctl@ceph.mon.b.smithi099.stdout: return self.__get_result() 2024-09-17T00:24:13.220 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/lib64/python3.9/concurrent/futures/_base.py", line 391, in __get_result 2024-09-17T00:24:13.220 INFO:journalctl@ceph.mon.b.smithi099.stdout: raise self._exception 2024-09-17T00:24:13.220 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1370, in _create_daemon 2024-09-17T00:24:13.220 INFO:journalctl@ceph.mon.b.smithi099.stdout: out, err, code = await self._run_cephadm( 2024-09-17T00:24:13.220 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1713, in _run_cephadm 2024-09-17T00:24:13.220 INFO:journalctl@ceph.mon.b.smithi099.stdout: raise OrchestratorError( 2024-09-17T00:24:13.220 INFO:journalctl@ceph.mon.b.smithi099.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi002.vmouem ... 2024-09-17T00:24:13.220 INFO:journalctl@ceph.mon.b.smithi099.stdout: Creating ceph-iscsi config... 2024-09-17T00:24:13.220 INFO:journalctl@ceph.mon.b.smithi099.stdout: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/iscsi-gateway.cfg 2024-09-17T00:24:13.221 INFO:journalctl@ceph.mon.b.smithi099.stdout: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/tcmu-runner-entrypoint.sh 2024-09-17T00:24:13.221 INFO:journalctl@ceph.mon.b.smithi099.stdout: Non-zero exit code 1 from systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem 2024-09-17T00:24:13.221 INFO:journalctl@ceph.mon.b.smithi099.stdout: systemctl: stderr Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:24:13.221 INFO:journalctl@ceph.mon.b.smithi099.stdout: systemctl: stderr See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:24:13.221 INFO:journalctl@ceph.mon.b.smithi099.stdout: Traceback (most recent call last): 2024-09-17T00:24:13.221 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-17T00:24:13.221 INFO:journalctl@ceph.mon.b.smithi099.stdout: return _run_code(code, main_globals, None, 2024-09-17T00:24:13.221 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-17T00:24:13.221 INFO:journalctl@ceph.mon.b.smithi099.stdout: exec(code, run_globals) 2024-09-17T00:24:13.221 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5579, in 2024-09-17T00:24:13.221 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5567, in main 2024-09-17T00:24:13.221 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3049, in command_deploy_from 2024-09-17T00:24:13.221 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3084, in _common_deploy 2024-09-17T00:24:13.222 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3104, in _deploy_daemon_container 2024-09-17T00:24:13.222 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 1124, in deploy_daemon 2024-09-17T00:24:13.222 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-17T00:24:13.222 INFO:journalctl@ceph.mon.b.smithi099.stdout: RuntimeError: Failed command: systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem: Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:24:13.222 INFO:journalctl@ceph.mon.b.smithi099.stdout: See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:24:13.222 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:12 smithi099 ceph-mon[24594]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T00:24:13.222 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:12 smithi099 ceph-mon[24594]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:24:13.222 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:12 smithi099 ceph-mon[24594]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-09-17T00:24:13.222 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:12 smithi099 ceph-mon[24594]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["b"]}]: dispatch 2024-09-17T00:24:13.222 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:12 smithi099 ceph-mon[24594]: Upgrade: It appears safe to stop mon.b 2024-09-17T00:24:13.222 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:12 smithi099 ceph-mon[24594]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 637 B/s rd, 0 op/s 2024-09-17T00:24:13.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:12 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:13.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:12 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T00:24:13.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:12 smithi002 ceph-mon[66732]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi002.vmouem ... 2024-09-17T00:24:13.229 INFO:journalctl@ceph.mon.c.smithi002.stdout: Creating ceph-iscsi config... 2024-09-17T00:24:13.229 INFO:journalctl@ceph.mon.c.smithi002.stdout: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/iscsi-gateway.cfg 2024-09-17T00:24:13.229 INFO:journalctl@ceph.mon.c.smithi002.stdout: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/tcmu-runner-entrypoint.sh 2024-09-17T00:24:13.229 INFO:journalctl@ceph.mon.c.smithi002.stdout: Non-zero exit code 1 from systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem 2024-09-17T00:24:13.229 INFO:journalctl@ceph.mon.c.smithi002.stdout: systemctl: stderr Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:24:13.230 INFO:journalctl@ceph.mon.c.smithi002.stdout: systemctl: stderr See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:24:13.230 INFO:journalctl@ceph.mon.c.smithi002.stdout: Traceback (most recent call last): 2024-09-17T00:24:13.230 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-17T00:24:13.230 INFO:journalctl@ceph.mon.c.smithi002.stdout: return _run_code(code, main_globals, None, 2024-09-17T00:24:13.230 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-17T00:24:13.230 INFO:journalctl@ceph.mon.c.smithi002.stdout: exec(code, run_globals) 2024-09-17T00:24:13.230 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5579, in 2024-09-17T00:24:13.230 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5567, in main 2024-09-17T00:24:13.230 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3049, in command_deploy_from 2024-09-17T00:24:13.230 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3084, in _common_deploy 2024-09-17T00:24:13.230 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3104, in _deploy_daemon_container 2024-09-17T00:24:13.230 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 1124, in deploy_daemon 2024-09-17T00:24:13.230 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-17T00:24:13.231 INFO:journalctl@ceph.mon.c.smithi002.stdout: RuntimeError: Failed command: systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem: Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:24:13.231 INFO:journalctl@ceph.mon.c.smithi002.stdout: See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:24:13.231 INFO:journalctl@ceph.mon.c.smithi002.stdout: Traceback (most recent call last): 2024-09-17T00:24:13.231 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1134, in _check_daemons 2024-09-17T00:24:13.231 INFO:journalctl@ceph.mon.c.smithi002.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-09-17T00:24:13.231 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2473, in _daemon_action 2024-09-17T00:24:13.231 INFO:journalctl@ceph.mon.c.smithi002.stdout: return self.wait_async( 2024-09-17T00:24:13.231 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 796, in wait_async 2024-09-17T00:24:13.231 INFO:journalctl@ceph.mon.c.smithi002.stdout: return self.event_loop.get_result(coro, timeout) 2024-09-17T00:24:13.231 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-09-17T00:24:13.231 INFO:journalctl@ceph.mon.c.smithi002.stdout: return future.result(timeout) 2024-09-17T00:24:13.231 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/lib64/python3.9/concurrent/futures/_base.py", line 446, in result 2024-09-17T00:24:13.231 INFO:journalctl@ceph.mon.c.smithi002.stdout: return self.__get_result() 2024-09-17T00:24:13.231 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/lib64/python3.9/concurrent/futures/_base.py", line 391, in __get_result 2024-09-17T00:24:13.232 INFO:journalctl@ceph.mon.c.smithi002.stdout: raise self._exception 2024-09-17T00:24:13.232 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1370, in _create_daemon 2024-09-17T00:24:13.232 INFO:journalctl@ceph.mon.c.smithi002.stdout: out, err, code = await self._run_cephadm( 2024-09-17T00:24:13.232 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1713, in _run_cephadm 2024-09-17T00:24:13.232 INFO:journalctl@ceph.mon.c.smithi002.stdout: raise OrchestratorError( 2024-09-17T00:24:13.232 INFO:journalctl@ceph.mon.c.smithi002.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi002.vmouem ... 2024-09-17T00:24:13.232 INFO:journalctl@ceph.mon.c.smithi002.stdout: Creating ceph-iscsi config... 2024-09-17T00:24:13.232 INFO:journalctl@ceph.mon.c.smithi002.stdout: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/iscsi-gateway.cfg 2024-09-17T00:24:13.232 INFO:journalctl@ceph.mon.c.smithi002.stdout: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/tcmu-runner-entrypoint.sh 2024-09-17T00:24:13.232 INFO:journalctl@ceph.mon.c.smithi002.stdout: Non-zero exit code 1 from systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem 2024-09-17T00:24:13.232 INFO:journalctl@ceph.mon.c.smithi002.stdout: systemctl: stderr Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:24:13.232 INFO:journalctl@ceph.mon.c.smithi002.stdout: systemctl: stderr See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:24:13.232 INFO:journalctl@ceph.mon.c.smithi002.stdout: Traceback (most recent call last): 2024-09-17T00:24:13.233 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-17T00:24:13.233 INFO:journalctl@ceph.mon.c.smithi002.stdout: return _run_code(code, main_globals, None, 2024-09-17T00:24:13.233 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-17T00:24:13.233 INFO:journalctl@ceph.mon.c.smithi002.stdout: exec(code, run_globals) 2024-09-17T00:24:13.233 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5579, in 2024-09-17T00:24:13.233 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5567, in main 2024-09-17T00:24:13.233 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3049, in command_deploy_from 2024-09-17T00:24:13.233 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3084, in _common_deploy 2024-09-17T00:24:13.233 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3104, in _deploy_daemon_container 2024-09-17T00:24:13.233 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 1124, in deploy_daemon 2024-09-17T00:24:13.233 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-17T00:24:13.233 INFO:journalctl@ceph.mon.c.smithi002.stdout: RuntimeError: Failed command: systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem: Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:24:13.233 INFO:journalctl@ceph.mon.c.smithi002.stdout: See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:24:13.234 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:12 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:13.234 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:12 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T00:24:13.234 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:12 smithi002 ceph-mon[63804]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi002.vmouem ... 2024-09-17T00:24:13.234 INFO:journalctl@ceph.mon.a.smithi002.stdout: Creating ceph-iscsi config... 2024-09-17T00:24:13.234 INFO:journalctl@ceph.mon.a.smithi002.stdout: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/iscsi-gateway.cfg 2024-09-17T00:24:13.235 INFO:journalctl@ceph.mon.a.smithi002.stdout: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/tcmu-runner-entrypoint.sh 2024-09-17T00:24:13.235 INFO:journalctl@ceph.mon.a.smithi002.stdout: Non-zero exit code 1 from systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem 2024-09-17T00:24:13.235 INFO:journalctl@ceph.mon.a.smithi002.stdout: systemctl: stderr Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:24:13.235 INFO:journalctl@ceph.mon.a.smithi002.stdout: systemctl: stderr See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:24:13.235 INFO:journalctl@ceph.mon.a.smithi002.stdout: Traceback (most recent call last): 2024-09-17T00:24:13.235 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-17T00:24:13.235 INFO:journalctl@ceph.mon.a.smithi002.stdout: return _run_code(code, main_globals, None, 2024-09-17T00:24:13.235 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-17T00:24:13.235 INFO:journalctl@ceph.mon.a.smithi002.stdout: exec(code, run_globals) 2024-09-17T00:24:13.235 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5579, in 2024-09-17T00:24:13.235 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5567, in main 2024-09-17T00:24:13.235 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3049, in command_deploy_from 2024-09-17T00:24:13.235 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3084, in _common_deploy 2024-09-17T00:24:13.236 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3104, in _deploy_daemon_container 2024-09-17T00:24:13.236 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 1124, in deploy_daemon 2024-09-17T00:24:13.236 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-17T00:24:13.236 INFO:journalctl@ceph.mon.a.smithi002.stdout: RuntimeError: Failed command: systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem: Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:24:13.236 INFO:journalctl@ceph.mon.a.smithi002.stdout: See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:24:13.236 INFO:journalctl@ceph.mon.a.smithi002.stdout: Traceback (most recent call last): 2024-09-17T00:24:13.236 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1134, in _check_daemons 2024-09-17T00:24:13.236 INFO:journalctl@ceph.mon.a.smithi002.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-09-17T00:24:13.236 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2473, in _daemon_action 2024-09-17T00:24:13.236 INFO:journalctl@ceph.mon.a.smithi002.stdout: return self.wait_async( 2024-09-17T00:24:13.236 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 796, in wait_async 2024-09-17T00:24:13.236 INFO:journalctl@ceph.mon.a.smithi002.stdout: return self.event_loop.get_result(coro, timeout) 2024-09-17T00:24:13.236 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-09-17T00:24:13.236 INFO:journalctl@ceph.mon.a.smithi002.stdout: return future.result(timeout) 2024-09-17T00:24:13.237 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/lib64/python3.9/concurrent/futures/_base.py", line 446, in result 2024-09-17T00:24:13.237 INFO:journalctl@ceph.mon.a.smithi002.stdout: return self.__get_result() 2024-09-17T00:24:13.237 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/lib64/python3.9/concurrent/futures/_base.py", line 391, in __get_result 2024-09-17T00:24:13.237 INFO:journalctl@ceph.mon.a.smithi002.stdout: raise self._exception 2024-09-17T00:24:13.237 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1370, in _create_daemon 2024-09-17T00:24:13.237 INFO:journalctl@ceph.mon.a.smithi002.stdout: out, err, code = await self._run_cephadm( 2024-09-17T00:24:13.237 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1713, in _run_cephadm 2024-09-17T00:24:13.237 INFO:journalctl@ceph.mon.a.smithi002.stdout: raise OrchestratorError( 2024-09-17T00:24:13.237 INFO:journalctl@ceph.mon.a.smithi002.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi002.vmouem ... 2024-09-17T00:24:13.237 INFO:journalctl@ceph.mon.a.smithi002.stdout: Creating ceph-iscsi config... 2024-09-17T00:24:13.237 INFO:journalctl@ceph.mon.a.smithi002.stdout: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/iscsi-gateway.cfg 2024-09-17T00:24:13.237 INFO:journalctl@ceph.mon.a.smithi002.stdout: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/tcmu-runner-entrypoint.sh 2024-09-17T00:24:13.237 INFO:journalctl@ceph.mon.a.smithi002.stdout: Non-zero exit code 1 from systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem 2024-09-17T00:24:13.237 INFO:journalctl@ceph.mon.a.smithi002.stdout: systemctl: stderr Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:24:13.238 INFO:journalctl@ceph.mon.a.smithi002.stdout: systemctl: stderr See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:24:13.238 INFO:journalctl@ceph.mon.a.smithi002.stdout: Traceback (most recent call last): 2024-09-17T00:24:13.238 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-17T00:24:13.238 INFO:journalctl@ceph.mon.a.smithi002.stdout: return _run_code(code, main_globals, None, 2024-09-17T00:24:13.238 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-17T00:24:13.238 INFO:journalctl@ceph.mon.a.smithi002.stdout: exec(code, run_globals) 2024-09-17T00:24:13.238 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5579, in 2024-09-17T00:24:13.238 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5567, in main 2024-09-17T00:24:13.238 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3049, in command_deploy_from 2024-09-17T00:24:13.238 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3084, in _common_deploy 2024-09-17T00:24:13.238 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3104, in _deploy_daemon_container 2024-09-17T00:24:13.238 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 1124, in deploy_daemon 2024-09-17T00:24:13.238 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-17T00:24:13.238 INFO:journalctl@ceph.mon.a.smithi002.stdout: RuntimeError: Failed command: systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem: Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:24:13.239 INFO:journalctl@ceph.mon.a.smithi002.stdout: See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:24:13.239 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:12 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T00:24:13.239 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:12 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:24:13.239 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:12 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-09-17T00:24:13.239 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:12 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["b"]}]: dispatch 2024-09-17T00:24:13.239 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:12 smithi002 ceph-mon[63804]: Upgrade: It appears safe to stop mon.b 2024-09-17T00:24:13.239 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:12 smithi002 ceph-mon[63804]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 637 B/s rd, 0 op/s 2024-09-17T00:24:13.239 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:12 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T00:24:13.239 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:12 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:24:13.239 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:12 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-09-17T00:24:13.239 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:12 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["b"]}]: dispatch 2024-09-17T00:24:13.239 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:12 smithi002 ceph-mon[66732]: Upgrade: It appears safe to stop mon.b 2024-09-17T00:24:13.240 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:12 smithi002 ceph-mon[66732]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 637 B/s rd, 0 op/s 2024-09-17T00:24:13.763 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:24:13 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[52757]: ts=2024-09-17T00:24:13.505Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes name=CephNodeDiskspaceWarning index=4 msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi002\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi002\", job=\"node\", machine=\"x86_64\", nodename=\"smithi002\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"3aad27a2-7489-11ef-bceb-c7b262605968\", domainname=\"(none)\", instance=\"smithi002\", job=\"node\", machine=\"x86_64\", nodename=\"smithi002\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-17T00:24:14.678 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:14 smithi099 ceph-mon[24594]: Upgrade: Updating mon.b 2024-09-17T00:24:14.678 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:14 smithi099 ceph-mon[24594]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:14.678 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:14 smithi099 ceph-mon[24594]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-17T00:24:14.678 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:14 smithi099 ceph-mon[24594]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-17T00:24:14.678 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:14 smithi099 ceph-mon[24594]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:24:14.678 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:14 smithi099 ceph-mon[24594]: Deploying daemon mon.b on smithi099 2024-09-17T00:24:14.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:14 smithi002 ceph-mon[66732]: Upgrade: Updating mon.b 2024-09-17T00:24:14.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:14 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:14.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:14 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-17T00:24:14.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:14 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-17T00:24:14.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:14 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:24:14.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:14 smithi002 ceph-mon[66732]: Deploying daemon mon.b on smithi099 2024-09-17T00:24:14.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:14 smithi002 ceph-mon[63804]: Upgrade: Updating mon.b 2024-09-17T00:24:14.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:14 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:14.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:14 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-17T00:24:14.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:14 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-17T00:24:14.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:14 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:24:14.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:14 smithi002 ceph-mon[63804]: Deploying daemon mon.b on smithi099 2024-09-17T00:24:15.124 INFO:teuthology.orchestra.run.smithi002.stdout:true 2024-09-17T00:24:15.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:14 smithi099 systemd[1]: Stopping Ceph mon.b for 3aad27a2-7489-11ef-bceb-c7b262605968... 2024-09-17T00:24:15.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:15 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mon-b[24577]: 2024-09-17T00:24:15.194+0000 7fdd99d1e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-17T00:24:15.218 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:15 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mon-b[24577]: 2024-09-17T00:24:15.194+0000 7fdd99d1e700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-09-17T00:24:15.479 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:24:15 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: [17/Sep/2024:00:24:15] ENGINE Bus STOPPING 2024-09-17T00:24:15.484 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:15 smithi099 podman[56554]: 2024-09-17 00:24:15.235759006 +0000 UTC m=+0.137099630 container died c6d2af7151872ee22f65b0ab7d833e966a67e7443f3369aa2a7ba9d93c7e9320 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mon-b, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, build-date=2022-05-03T08:36:31.336870, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, maintainer=Guillaume Abrioux , io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, ceph=True, vendor=Red Hat, Inc., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, release=754, version=8, com.redhat.component=centos-stream-container, RELEASE=HEAD, GIT_CLEAN=True, io.buildah.version=1.19.8, GIT_BRANCH=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-type=git, io.k8s.display-name=CentOS Stream 8, io.openshift.tags=base centos centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, architecture=x86_64, io.openshift.expose-services=, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, CEPH_POINT_RELEASE=-17.2.0, name=centos-stream) 2024-09-17T00:24:15.767 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:15 smithi099 podman[56554]: 2024-09-17 00:24:15.577695041 +0000 UTC m=+0.479035664 container cleanup c6d2af7151872ee22f65b0ab7d833e966a67e7443f3369aa2a7ba9d93c7e9320 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mon-b, vcs-type=git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.display-name=CentOS Stream 8, GIT_BRANCH=HEAD, name=centos-stream, build-date=2022-05-03T08:36:31.336870, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vendor=Red Hat, Inc., CEPH_POINT_RELEASE=-17.2.0, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, maintainer=Guillaume Abrioux , RELEASE=HEAD, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=centos-stream-container, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, release=754, io.openshift.tags=base centos centos-stream, ceph=True, distribution-scope=public, GIT_CLEAN=True, version=8, io.buildah.version=1.19.8, io.openshift.expose-services=) 2024-09-17T00:24:15.767 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:15 smithi099 bash[56554]: ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mon-b 2024-09-17T00:24:15.818 INFO:teuthology.orchestra.run.smithi002.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-17T00:24:15.818 INFO:teuthology.orchestra.run.smithi002.stdout:alertmanager.a smithi002 *:9093,9094 running (2m) 11s ago 6m 18.8M - 0.25.0 c8568f914cd2 a21e13b3d677 2024-09-17T00:24:15.818 INFO:teuthology.orchestra.run.smithi002.stdout:grafana.a smithi099 *:3000 running (99s) 29s ago 6m 43.2M - 8.3.5 dad864ee21e9 bafb824a3c08 2024-09-17T00:24:15.818 INFO:teuthology.orchestra.run.smithi002.stdout:iscsi.foo.smithi002.vmouem smithi002 unknown 11s ago 5m - - 2024-09-17T00:24:15.818 INFO:teuthology.orchestra.run.smithi002.stdout:mgr.x smithi099 *:8443,9283,8765 running (59s) 29s ago 11m 437M - 19.1.1-330-gc2ddf972 ebb0615149de 6aa2ccef9ca0 2024-09-17T00:24:15.818 INFO:teuthology.orchestra.run.smithi002.stdout:mgr.y smithi002 *:8443,9283,8765 running (93s) 11s ago 13m 492M - 19.1.1-330-gc2ddf972 ebb0615149de 8838897e8449 2024-09-17T00:24:15.818 INFO:teuthology.orchestra.run.smithi002.stdout:mon.a smithi002 running (42s) 11s ago 13m 39.9M 2048M 19.1.1-330-gc2ddf972 ebb0615149de a035def53ef6 2024-09-17T00:24:15.819 INFO:teuthology.orchestra.run.smithi002.stdout:mon.b smithi099 running (11m) 29s ago 11m 54.9M 2048M 17.2.0 e1d6a67b021e c6d2af715187 2024-09-17T00:24:15.819 INFO:teuthology.orchestra.run.smithi002.stdout:mon.c smithi002 running (17s) 11s ago 12m 18.5M 2048M 19.1.1-330-gc2ddf972 ebb0615149de 8a510255ae66 2024-09-17T00:24:15.819 INFO:teuthology.orchestra.run.smithi002.stdout:node-exporter.a smithi002 *:9100 running (2m) 11s ago 7m 15.0M - 1.5.0 0da6a335fe13 1030ae01bb32 2024-09-17T00:24:15.819 INFO:teuthology.orchestra.run.smithi002.stdout:node-exporter.b smithi099 *:9100 running (103s) 29s ago 7m 16.8M - 1.5.0 0da6a335fe13 01fcd35dca8a 2024-09-17T00:24:15.819 INFO:teuthology.orchestra.run.smithi002.stdout:osd.0 smithi002 running (11m) 11s ago 11m 53.6M 2737M 17.2.0 e1d6a67b021e a0ad0fb7aa34 2024-09-17T00:24:15.819 INFO:teuthology.orchestra.run.smithi002.stdout:osd.1 smithi002 running (10m) 11s ago 10m 56.0M 2737M 17.2.0 e1d6a67b021e 7cb62c311c7d 2024-09-17T00:24:15.819 INFO:teuthology.orchestra.run.smithi002.stdout:osd.2 smithi002 running (10m) 11s ago 10m 52.5M 2737M 17.2.0 e1d6a67b021e eafddacb955e 2024-09-17T00:24:15.819 INFO:teuthology.orchestra.run.smithi002.stdout:osd.3 smithi002 running (10m) 11s ago 10m 55.4M 2737M 17.2.0 e1d6a67b021e fd99d3764412 2024-09-17T00:24:15.819 INFO:teuthology.orchestra.run.smithi002.stdout:osd.4 smithi099 running (9m) 29s ago 9m 56.0M 2992M 17.2.0 e1d6a67b021e 0af1827b658d 2024-09-17T00:24:15.819 INFO:teuthology.orchestra.run.smithi002.stdout:osd.5 smithi099 running (9m) 29s ago 9m 54.7M 2992M 17.2.0 e1d6a67b021e 4f7ef3aefc58 2024-09-17T00:24:15.819 INFO:teuthology.orchestra.run.smithi002.stdout:osd.6 smithi099 running (8m) 29s ago 8m 50.2M 2992M 17.2.0 e1d6a67b021e 3213f5e6952a 2024-09-17T00:24:15.819 INFO:teuthology.orchestra.run.smithi002.stdout:osd.7 smithi099 running (8m) 29s ago 8m 56.8M 2992M 17.2.0 e1d6a67b021e 63be3a1ed0a1 2024-09-17T00:24:15.819 INFO:teuthology.orchestra.run.smithi002.stdout:prometheus.a smithi099 *:9095 running (64s) 29s ago 7m 46.1M - 2.43.0 a07b618ecd1d bf32c2ae11dc 2024-09-17T00:24:15.820 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.foo.smithi002.uwdhcs smithi002 *:8000 running (6m) 11s ago 6m 75.3M - 17.2.0 e1d6a67b021e 9c35813f459a 2024-09-17T00:24:15.820 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.foo.smithi099.utzfap smithi099 *:8000 running (6m) 29s ago 6m 75.6M - 17.2.0 e1d6a67b021e 34ec989160a6 2024-09-17T00:24:15.820 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.smpl.smithi002.rqnxii smithi002 *:80 running (6m) 11s ago 6m 73.6M - 17.2.0 e1d6a67b021e 0e180d0afb99 2024-09-17T00:24:15.820 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.smpl.smithi099.nzvkpm smithi099 *:80 running (6m) 29s ago 5m 76.1M - 17.2.0 e1d6a67b021e 6a068e82ac72 2024-09-17T00:24:15.979 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:24:15 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: [17/Sep/2024:00:24:15] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-17T00:24:15.979 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:24:15 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: [17/Sep/2024:00:24:15] ENGINE Bus STOPPED 2024-09-17T00:24:15.979 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:24:15 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: [17/Sep/2024:00:24:15] ENGINE Bus STARTING 2024-09-17T00:24:15.979 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:24:15 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: [17/Sep/2024:00:24:15] ENGINE Serving on http://:::9283 2024-09-17T00:24:15.979 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:24:15 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: [17/Sep/2024:00:24:15] ENGINE Bus STARTED 2024-09-17T00:24:16.189 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:15 smithi099 podman[56568]: 2024-09-17 00:24:15.770520178 +0000 UTC m=+0.532364011 container remove c6d2af7151872ee22f65b0ab7d833e966a67e7443f3369aa2a7ba9d93c7e9320 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mon-b, version=8, io.openshift.expose-services=, distribution-scope=public, CEPH_POINT_RELEASE=-17.2.0, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, name=centos-stream, GIT_CLEAN=True, io.buildah.version=1.19.8, maintainer=Guillaume Abrioux , io.openshift.tags=base centos centos-stream, com.redhat.component=centos-stream-container, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-type=git, GIT_BRANCH=HEAD, io.k8s.display-name=CentOS Stream 8, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, build-date=2022-05-03T08:36:31.336870, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vendor=Red Hat, Inc., RELEASE=HEAD, ceph=True, release=754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb) 2024-09-17T00:24:16.189 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:15 smithi099 systemd[1]: ceph-3aad27a2-7489-11ef-bceb-c7b262605968@mon.b.service: Deactivated successfully. 2024-09-17T00:24:16.189 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:15 smithi099 systemd[1]: Stopped Ceph mon.b for 3aad27a2-7489-11ef-bceb-c7b262605968. 2024-09-17T00:24:16.189 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:15 smithi099 systemd[1]: ceph-3aad27a2-7489-11ef-bceb-c7b262605968@mon.b.service: Consumed 9.701s CPU time. 2024-09-17T00:24:16.233 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-09-17T00:24:16.233 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": { 2024-09-17T00:24:16.233 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-09-17T00:24:16.233 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 19.1.1-330-gc2ddf972 (c2ddf9721ef254645201cdbabd9c8f49c30a586f) squid (stable)": 2 2024-09-17T00:24:16.233 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-09-17T00:24:16.233 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": { 2024-09-17T00:24:16.233 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 19.1.1-330-gc2ddf972 (c2ddf9721ef254645201cdbabd9c8f49c30a586f) squid (stable)": 2 2024-09-17T00:24:16.233 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-09-17T00:24:16.234 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": { 2024-09-17T00:24:16.234 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-09-17T00:24:16.234 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-09-17T00:24:16.234 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw": { 2024-09-17T00:24:16.234 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-09-17T00:24:16.234 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-09-17T00:24:16.234 INFO:teuthology.orchestra.run.smithi002.stdout: "overall": { 2024-09-17T00:24:16.234 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 13, 2024-09-17T00:24:16.234 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 19.1.1-330-gc2ddf972 (c2ddf9721ef254645201cdbabd9c8f49c30a586f) squid (stable)": 4 2024-09-17T00:24:16.234 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-09-17T00:24:16.234 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-09-17T00:24:16.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 systemd[1]: Starting Ceph mon.b for 3aad27a2-7489-11ef-bceb-c7b262605968... 2024-09-17T00:24:16.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 podman[56693]: 2024-09-17 00:24:16.304185812 +0000 UTC m=+0.016494106 image pull ebb0615149def4a3179c82c0cc47af75808300abb9f1142788e5e605a7ac0e5b quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f 2024-09-17T00:24:16.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 podman[56693]: 2024-09-17 00:24:16.413476763 +0000 UTC m=+0.125785049 container create aab3ab73a850570e9284d848e27b6d1efd9bba4063d67a8b5aa8413115306edf (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mon-b, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=squid-c2ddf97) 2024-09-17T00:24:16.585 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-09-17T00:24:16.585 INFO:teuthology.orchestra.run.smithi002.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f", 2024-09-17T00:24:16.585 INFO:teuthology.orchestra.run.smithi002.stdout: "in_progress": true, 2024-09-17T00:24:16.585 INFO:teuthology.orchestra.run.smithi002.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-17T00:24:16.585 INFO:teuthology.orchestra.run.smithi002.stdout: "services_complete": [ 2024-09-17T00:24:16.586 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr" 2024-09-17T00:24:16.586 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-09-17T00:24:16.586 INFO:teuthology.orchestra.run.smithi002.stdout: "progress": "4/23 daemons upgraded", 2024-09-17T00:24:16.586 INFO:teuthology.orchestra.run.smithi002.stdout: "message": "Currently upgrading mon daemons", 2024-09-17T00:24:16.586 INFO:teuthology.orchestra.run.smithi002.stdout: "is_paused": false 2024-09-17T00:24:16.586 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-09-17T00:24:16.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 podman[56693]: 2024-09-17 00:24:16.682551909 +0000 UTC m=+0.394860199 container init aab3ab73a850570e9284d848e27b6d1efd9bba4063d67a8b5aa8413115306edf (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mon-b, GIT_CLEAN=True, RELEASE=squid-c2ddf97, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS) 2024-09-17T00:24:16.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 podman[56693]: 2024-09-17 00:24:16.685995006 +0000 UTC m=+0.398303302 container start aab3ab73a850570e9284d848e27b6d1efd9bba4063d67a8b5aa8413115306edf (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mon-b, org.label-schema.build-date=20240731, RELEASE=squid-c2ddf97, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True) 2024-09-17T00:24:16.968 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: set uid:gid to 167:167 (ceph:ceph) 2024-09-17T00:24:16.968 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: ceph version 19.1.1-330-gc2ddf972 (c2ddf9721ef254645201cdbabd9c8f49c30a586f) squid (stable), process ceph-mon, pid 2 2024-09-17T00:24:16.968 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: pidfile_write: ignore empty --pid-file 2024-09-17T00:24:16.968 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: load: jerasure load: lrc 2024-09-17T00:24:16.968 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: RocksDB version: 7.9.2 2024-09-17T00:24:16.968 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Git sha 0 2024-09-17T00:24:16.968 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Compile date 2024-09-16 15:09:42 2024-09-17T00:24:16.968 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: DB SUMMARY 2024-09-17T00:24:16.968 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: DB Session ID: LUJ3GWD8CFQXZIJKWR5J 2024-09-17T00:24:16.968 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: CURRENT file: CURRENT 2024-09-17T00:24:16.968 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: IDENTITY file: IDENTITY 2024-09-17T00:24:16.968 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: MANIFEST file: MANIFEST-000009 size: 1084 Bytes 2024-09-17T00:24:16.968 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 1, files: 000027.sst 2024-09-17T00:24:16.969 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000025.log size: 1397466 ; 2024-09-17T00:24:16.969 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.error_if_exists: 0 2024-09-17T00:24:16.969 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.create_if_missing: 0 2024-09-17T00:24:16.969 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.paranoid_checks: 1 2024-09-17T00:24:16.969 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.flush_verify_memtable_count: 1 2024-09-17T00:24:16.969 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-17T00:24:16.969 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-09-17T00:24:16.969 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.env: 0x55d8a53d1940 2024-09-17T00:24:16.969 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.fs: PosixFileSystem 2024-09-17T00:24:16.969 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.info_log: 0x55d8a68956e0 2024-09-17T00:24:16.969 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.max_file_opening_threads: 16 2024-09-17T00:24:16.969 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.statistics: (nil) 2024-09-17T00:24:16.969 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.use_fsync: 0 2024-09-17T00:24:16.970 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.max_log_file_size: 0 2024-09-17T00:24:16.970 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-17T00:24:16.970 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-17T00:24:16.970 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.keep_log_file_num: 1000 2024-09-17T00:24:16.970 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.recycle_log_file_num: 0 2024-09-17T00:24:16.970 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.allow_fallocate: 1 2024-09-17T00:24:16.970 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.allow_mmap_reads: 0 2024-09-17T00:24:16.970 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.allow_mmap_writes: 0 2024-09-17T00:24:16.970 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.use_direct_reads: 0 2024-09-17T00:24:16.970 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-17T00:24:16.970 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.create_missing_column_families: 0 2024-09-17T00:24:16.970 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.db_log_dir: 2024-09-17T00:24:16.970 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.wal_dir: 2024-09-17T00:24:16.970 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-17T00:24:16.970 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-17T00:24:16.971 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-17T00:24:16.971 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-17T00:24:16.971 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-17T00:24:16.971 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-17T00:24:16.971 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.advise_random_on_open: 1 2024-09-17T00:24:16.971 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.db_write_buffer_size: 0 2024-09-17T00:24:16.971 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.write_buffer_manager: 0x55d8a68a4dc0 2024-09-17T00:24:16.971 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-17T00:24:16.971 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-17T00:24:16.971 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-17T00:24:16.971 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.rate_limiter: (nil) 2024-09-17T00:24:16.971 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-17T00:24:16.971 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.wal_recovery_mode: 2 2024-09-17T00:24:16.971 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.enable_thread_tracking: 0 2024-09-17T00:24:16.972 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.enable_pipelined_write: 0 2024-09-17T00:24:16.972 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.unordered_write: 0 2024-09-17T00:24:16.972 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-17T00:24:16.972 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-17T00:24:16.972 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-17T00:24:16.972 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-17T00:24:16.972 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.row_cache: None 2024-09-17T00:24:16.972 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.wal_filter: None 2024-09-17T00:24:16.972 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-17T00:24:16.974 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.allow_ingest_behind: 0 2024-09-17T00:24:16.974 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.two_write_queues: 0 2024-09-17T00:24:16.974 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.manual_wal_flush: 0 2024-09-17T00:24:16.974 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.wal_compression: 0 2024-09-17T00:24:16.974 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.atomic_flush: 0 2024-09-17T00:24:16.974 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-17T00:24:16.974 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-17T00:24:16.974 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-17T00:24:16.974 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.log_readahead_size: 0 2024-09-17T00:24:16.974 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-17T00:24:16.974 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.best_efforts_recovery: 0 2024-09-17T00:24:16.974 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-17T00:24:16.974 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-17T00:24:16.975 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.allow_data_in_errors: 0 2024-09-17T00:24:16.975 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.db_host_id: __hostname__ 2024-09-17T00:24:16.975 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.enforce_single_del_contracts: true 2024-09-17T00:24:16.975 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.max_background_jobs: 2 2024-09-17T00:24:16.975 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.max_background_compactions: -1 2024-09-17T00:24:16.975 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.max_subcompactions: 1 2024-09-17T00:24:16.975 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-17T00:24:16.975 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-17T00:24:16.975 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-17T00:24:16.975 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.max_total_wal_size: 0 2024-09-17T00:24:16.975 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-17T00:24:16.975 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-17T00:24:16.975 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-17T00:24:16.975 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-17T00:24:16.975 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.max_open_files: -1 2024-09-17T00:24:16.976 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.bytes_per_sync: 0 2024-09-17T00:24:16.976 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-17T00:24:16.976 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-17T00:24:16.976 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.compaction_readahead_size: 0 2024-09-17T00:24:16.976 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.max_background_flushes: -1 2024-09-17T00:24:16.976 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Compression algorithms supported: 2024-09-17T00:24:16.976 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: kZSTD supported: 0 2024-09-17T00:24:16.976 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: kXpressCompression supported: 0 2024-09-17T00:24:16.976 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: kBZip2Compression supported: 0 2024-09-17T00:24:16.976 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-17T00:24:16.976 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: kLZ4Compression supported: 1 2024-09-17T00:24:16.976 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: kZlibCompression supported: 1 2024-09-17T00:24:16.976 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: kLZ4HCCompression supported: 1 2024-09-17T00:24:16.976 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: kSnappyCompression supported: 1 2024-09-17T00:24:16.976 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-17T00:24:16.977 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: DMutex implementation: pthread_mutex_t 2024-09-17T00:24:16.977 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000009 2024-09-17T00:24:16.977 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-09-17T00:24:16.977 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-17T00:24:16.977 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.merge_operator: 2024-09-17T00:24:16.977 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.compaction_filter: None 2024-09-17T00:24:16.977 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.compaction_filter_factory: None 2024-09-17T00:24:16.977 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.sst_partitioner_factory: None 2024-09-17T00:24:16.977 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-17T00:24:16.977 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-17T00:24:16.977 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55d8a6895320) 2024-09-17T00:24:16.977 INFO:journalctl@ceph.mon.b.smithi099.stdout: cache_index_and_filter_blocks: 1 2024-09-17T00:24:16.977 INFO:journalctl@ceph.mon.b.smithi099.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-17T00:24:16.977 INFO:journalctl@ceph.mon.b.smithi099.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-17T00:24:16.978 INFO:journalctl@ceph.mon.b.smithi099.stdout: pin_top_level_index_and_filter: 1 2024-09-17T00:24:16.978 INFO:journalctl@ceph.mon.b.smithi099.stdout: index_type: 0 2024-09-17T00:24:16.978 INFO:journalctl@ceph.mon.b.smithi099.stdout: data_block_index_type: 0 2024-09-17T00:24:16.978 INFO:journalctl@ceph.mon.b.smithi099.stdout: index_shortening: 1 2024-09-17T00:24:16.978 INFO:journalctl@ceph.mon.b.smithi099.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-17T00:24:16.978 INFO:journalctl@ceph.mon.b.smithi099.stdout: checksum: 4 2024-09-17T00:24:16.978 INFO:journalctl@ceph.mon.b.smithi099.stdout: no_block_cache: 0 2024-09-17T00:24:16.978 INFO:journalctl@ceph.mon.b.smithi099.stdout: block_cache: 0x55d8a68891f0 2024-09-17T00:24:16.978 INFO:journalctl@ceph.mon.b.smithi099.stdout: block_cache_name: BinnedLRUCache 2024-09-17T00:24:16.978 INFO:journalctl@ceph.mon.b.smithi099.stdout: block_cache_options: 2024-09-17T00:24:16.978 INFO:journalctl@ceph.mon.b.smithi099.stdout: capacity : 536870912 2024-09-17T00:24:16.978 INFO:journalctl@ceph.mon.b.smithi099.stdout: num_shard_bits : 4 2024-09-17T00:24:16.978 INFO:journalctl@ceph.mon.b.smithi099.stdout: strict_capacity_limit : 0 2024-09-17T00:24:16.979 INFO:journalctl@ceph.mon.b.smithi099.stdout: high_pri_pool_ratio: 0.000 2024-09-17T00:24:16.979 INFO:journalctl@ceph.mon.b.smithi099.stdout: block_cache_compressed: (nil) 2024-09-17T00:24:16.979 INFO:journalctl@ceph.mon.b.smithi099.stdout: persistent_cache: (nil) 2024-09-17T00:24:16.979 INFO:journalctl@ceph.mon.b.smithi099.stdout: block_size: 4096 2024-09-17T00:24:16.979 INFO:journalctl@ceph.mon.b.smithi099.stdout: block_size_deviation: 10 2024-09-17T00:24:16.979 INFO:journalctl@ceph.mon.b.smithi099.stdout: block_restart_interval: 16 2024-09-17T00:24:16.979 INFO:journalctl@ceph.mon.b.smithi099.stdout: index_block_restart_interval: 1 2024-09-17T00:24:16.979 INFO:journalctl@ceph.mon.b.smithi099.stdout: metadata_block_size: 4096 2024-09-17T00:24:16.979 INFO:journalctl@ceph.mon.b.smithi099.stdout: partition_filters: 0 2024-09-17T00:24:16.979 INFO:journalctl@ceph.mon.b.smithi099.stdout: use_delta_encoding: 1 2024-09-17T00:24:16.979 INFO:journalctl@ceph.mon.b.smithi099.stdout: filter_policy: bloomfilter 2024-09-17T00:24:16.979 INFO:journalctl@ceph.mon.b.smithi099.stdout: whole_key_filtering: 1 2024-09-17T00:24:16.979 INFO:journalctl@ceph.mon.b.smithi099.stdout: verify_compression: 0 2024-09-17T00:24:16.979 INFO:journalctl@ceph.mon.b.smithi099.stdout: read_amp_bytes_per_bit: 0 2024-09-17T00:24:16.980 INFO:journalctl@ceph.mon.b.smithi099.stdout: format_version: 5 2024-09-17T00:24:16.980 INFO:journalctl@ceph.mon.b.smithi099.stdout: enable_index_compression: 1 2024-09-17T00:24:16.980 INFO:journalctl@ceph.mon.b.smithi099.stdout: block_align: 0 2024-09-17T00:24:16.980 INFO:journalctl@ceph.mon.b.smithi099.stdout: max_auto_readahead_size: 262144 2024-09-17T00:24:16.980 INFO:journalctl@ceph.mon.b.smithi099.stdout: prepopulate_block_cache: 0 2024-09-17T00:24:16.980 INFO:journalctl@ceph.mon.b.smithi099.stdout: initial_auto_readahead_size: 8192 2024-09-17T00:24:16.980 INFO:journalctl@ceph.mon.b.smithi099.stdout: num_file_reads_for_auto_readahead: 2 2024-09-17T00:24:16.980 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.write_buffer_size: 33554432 2024-09-17T00:24:16.980 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.max_write_buffer_number: 2 2024-09-17T00:24:16.980 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.compression: NoCompression 2024-09-17T00:24:16.980 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.bottommost_compression: Disabled 2024-09-17T00:24:16.980 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.prefix_extractor: nullptr 2024-09-17T00:24:16.980 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-17T00:24:16.980 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.num_levels: 7 2024-09-17T00:24:16.981 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-17T00:24:16.981 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-17T00:24:16.981 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-17T00:24:16.981 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-17T00:24:16.981 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-17T00:24:16.981 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-17T00:24:16.981 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-17T00:24:16.981 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-17T00:24:16.981 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-17T00:24:16.981 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-17T00:24:16.981 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-09-17T00:24:16.981 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-09-17T00:24:16.981 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-17T00:24:16.981 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.compression_opts.level: 32767 2024-09-17T00:24:16.981 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.compression_opts.strategy: 0 2024-09-17T00:24:16.982 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-17T00:24:16.982 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-17T00:24:16.982 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-09-17T00:24:16.982 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-17T00:24:16.982 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.compression_opts.enabled: false 2024-09-17T00:24:16.982 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-09-17T00:24:16.982 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-17T00:24:16.982 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-17T00:24:16.982 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-17T00:24:16.982 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.target_file_size_base: 67108864 2024-09-17T00:24:16.982 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-17T00:24:16.982 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-17T00:24:16.982 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-17T00:24:16.982 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-17T00:24:16.983 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-17T00:24:16.983 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-17T00:24:16.983 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-17T00:24:16.983 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-17T00:24:16.983 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-17T00:24:16.983 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-17T00:24:16.983 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-17T00:24:16.983 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-17T00:24:16.983 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-17T00:24:16.983 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-09-17T00:24:16.983 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.arena_block_size: 1048576 2024-09-17T00:24:16.983 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-17T00:24:16.983 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-17T00:24:16.983 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.disable_auto_compactions: 0 2024-09-17T00:24:16.984 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-17T00:24:16.984 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-17T00:24:16.984 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-17T00:24:16.984 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-17T00:24:16.984 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-17T00:24:16.984 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-17T00:24:16.984 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-17T00:24:16.984 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-17T00:24:16.984 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-17T00:24:16.984 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-17T00:24:16.984 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-17T00:24:16.984 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.inplace_update_support: 0 2024-09-17T00:24:16.984 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-17T00:24:16.984 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-17T00:24:16.984 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-17T00:24:16.985 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-17T00:24:16.985 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.bloom_locality: 0 2024-09-17T00:24:16.985 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.max_successive_merges: 0 2024-09-17T00:24:16.985 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-17T00:24:16.985 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.paranoid_file_checks: 0 2024-09-17T00:24:16.985 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.force_consistency_checks: 1 2024-09-17T00:24:16.985 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.report_bg_io_stats: 0 2024-09-17T00:24:16.985 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.ttl: 2592000 2024-09-17T00:24:16.985 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-17T00:24:16.985 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-09-17T00:24:16.985 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-09-17T00:24:16.985 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.enable_blob_files: false 2024-09-17T00:24:16.985 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.min_blob_size: 0 2024-09-17T00:24:16.986 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.blob_file_size: 268435456 2024-09-17T00:24:16.986 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-17T00:24:16.986 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-17T00:24:16.986 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-17T00:24:16.986 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-09-17T00:24:16.986 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-09-17T00:24:16.986 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.blob_file_starting_level: 0 2024-09-17T00:24:16.986 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-09-17T00:24:16.986 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 27.sst 2024-09-17T00:24:16.986 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-09-17T00:24:16.986 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 29, last_sequence is 15029, log_number is 25,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-09-17T00:24:16.986 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 25 2024-09-17T00:24:16.986 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 471f84f1-b3a3-4dc9-8118-4b47ca9953b6 2024-09-17T00:24:16.986 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726532656712672, "job": 1, "event": "recovery_started", "wal_files": [25]} 2024-09-17T00:24:16.986 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #25 mode 2 2024-09-17T00:24:16.987 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726532656716924, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 30, "file_size": 852885, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 15034, "largest_seqno": 15852, "table_properties": {"data_size": 848746, "index_size": 1754, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 1349, "raw_key_size": 10691, "raw_average_key_size": 20, "raw_value_size": 839708, "raw_average_value_size": 1633, "num_data_blocks": 78, "num_entries": 514, "num_filter_entries": 514, "num_deletions": 251, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1726532656, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "471f84f1-b3a3-4dc9-8118-4b47ca9953b6", "db_session_id": "LUJ3GWD8CFQXZIJKWR5J", "orig_file_number": 30, "seqno_to_time_mapping": "N/A"}} 2024-09-17T00:24:16.987 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726532656717040, "job": 1, "event": "recovery_finished"} 2024-09-17T00:24:16.987 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: [db/version_set.cc:5047] Creating manifest 32 2024-09-17T00:24:16.987 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-09-17T00:24:16.987 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000025.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-17T00:24:16.987 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55d8a68b6e00 2024-09-17T00:24:16.987 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: DB pointer 0x55d8a69b2000 2024-09-17T00:24:16.987 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-09-17T00:24:16.987 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-09-17T00:24:16.987 INFO:journalctl@ceph.mon.b.smithi099.stdout: ** DB Stats ** 2024-09-17T00:24:16.987 INFO:journalctl@ceph.mon.b.smithi099.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-17T00:24:16.987 INFO:journalctl@ceph.mon.b.smithi099.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-17T00:24:16.987 INFO:journalctl@ceph.mon.b.smithi099.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-17T00:24:16.987 INFO:journalctl@ceph.mon.b.smithi099.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-17T00:24:16.988 INFO:journalctl@ceph.mon.b.smithi099.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-17T00:24:16.988 INFO:journalctl@ceph.mon.b.smithi099.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-17T00:24:16.988 INFO:journalctl@ceph.mon.b.smithi099.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-17T00:24:16.988 INFO:journalctl@ceph.mon.b.smithi099.stdout: 2024-09-17T00:24:16.988 INFO:journalctl@ceph.mon.b.smithi099.stdout: ** Compaction Stats [default] ** 2024-09-17T00:24:16.988 INFO:journalctl@ceph.mon.b.smithi099.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-17T00:24:16.988 INFO:journalctl@ceph.mon.b.smithi099.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-09-17T00:24:16.988 INFO:journalctl@ceph.mon.b.smithi099.stdout: L0 1/0 832.90 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 368.2 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-09-17T00:24:16.988 INFO:journalctl@ceph.mon.b.smithi099.stdout: L6 1/0 11.87 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-09-17T00:24:16.988 INFO:journalctl@ceph.mon.b.smithi099.stdout: Sum 2/0 12.68 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 368.2 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-09-17T00:24:16.988 INFO:journalctl@ceph.mon.b.smithi099.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 368.2 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-09-17T00:24:16.988 INFO:journalctl@ceph.mon.b.smithi099.stdout: 2024-09-17T00:24:16.988 INFO:journalctl@ceph.mon.b.smithi099.stdout: ** Compaction Stats [default] ** 2024-09-17T00:24:16.989 INFO:journalctl@ceph.mon.b.smithi099.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-17T00:24:16.989 INFO:journalctl@ceph.mon.b.smithi099.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-17T00:24:16.989 INFO:journalctl@ceph.mon.b.smithi099.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 368.2 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-09-17T00:24:16.989 INFO:journalctl@ceph.mon.b.smithi099.stdout: 2024-09-17T00:24:16.989 INFO:journalctl@ceph.mon.b.smithi099.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-09-17T00:24:16.989 INFO:journalctl@ceph.mon.b.smithi099.stdout: 2024-09-17T00:24:16.989 INFO:journalctl@ceph.mon.b.smithi099.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-17T00:24:16.989 INFO:journalctl@ceph.mon.b.smithi099.stdout: Flush(GB): cumulative 0.001, interval 0.001 2024-09-17T00:24:16.989 INFO:journalctl@ceph.mon.b.smithi099.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-17T00:24:16.989 INFO:journalctl@ceph.mon.b.smithi099.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-17T00:24:16.989 INFO:journalctl@ceph.mon.b.smithi099.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-17T00:24:16.989 INFO:journalctl@ceph.mon.b.smithi099.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-17T00:24:16.989 INFO:journalctl@ceph.mon.b.smithi099.stdout: Cumulative compaction: 0.00 GB write, 126.22 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-17T00:24:16.990 INFO:journalctl@ceph.mon.b.smithi099.stdout: Interval compaction: 0.00 GB write, 126.22 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-17T00:24:16.990 INFO:journalctl@ceph.mon.b.smithi099.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-17T00:24:16.990 INFO:journalctl@ceph.mon.b.smithi099.stdout: Block cache BinnedLRUCache@0x55d8a68891f0#2 capacity: 512.00 MB usage: 3.23 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-09-17T00:24:16.990 INFO:journalctl@ceph.mon.b.smithi099.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,1.41 KB,0.000268221%) IndexBlock(1,1.83 KB,0.000348687%) Misc(1,0.00 KB,0%) 2024-09-17T00:24:16.990 INFO:journalctl@ceph.mon.b.smithi099.stdout: 2024-09-17T00:24:16.990 INFO:journalctl@ceph.mon.b.smithi099.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-17T00:24:16.990 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: starting mon.b rank 2 at public addrs [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] at bind addrs [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid 3aad27a2-7489-11ef-bceb-c7b262605968 2024-09-17T00:24:16.990 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: mon.b@-1(???) e3 preinit fsid 3aad27a2-7489-11ef-bceb-c7b262605968 2024-09-17T00:24:16.990 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: mon.b@-1(???).mds e1 new map 2024-09-17T00:24:16.990 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: mon.b@-1(???).mds e1 print_map 2024-09-17T00:24:16.990 INFO:journalctl@ceph.mon.b.smithi099.stdout: e1 2024-09-17T00:24:16.990 INFO:journalctl@ceph.mon.b.smithi099.stdout: btime 1970-01-01T00:00:00:000000+0000 2024-09-17T00:24:16.990 INFO:journalctl@ceph.mon.b.smithi099.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-17T00:24:16.991 INFO:journalctl@ceph.mon.b.smithi099.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-09-17T00:24:16.991 INFO:journalctl@ceph.mon.b.smithi099.stdout: legacy client fscid: -1 2024-09-17T00:24:16.991 INFO:journalctl@ceph.mon.b.smithi099.stdout: 2024-09-17T00:24:16.991 INFO:journalctl@ceph.mon.b.smithi099.stdout: No filesystems configured 2024-09-17T00:24:16.991 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: mon.b@-1(???).osd e110 crush map has features 3314933000854323200, adjusting msgr requires 2024-09-17T00:24:16.991 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: mon.b@-1(???).osd e110 crush map has features 432629239337189376, adjusting msgr requires 2024-09-17T00:24:16.991 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: mon.b@-1(???).osd e110 crush map has features 432629239337189376, adjusting msgr requires 2024-09-17T00:24:16.991 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: mon.b@-1(???).osd e110 crush map has features 432629239337189376, adjusting msgr requires 2024-09-17T00:24:16.991 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 ceph-mon[56712]: mon.b@-1(???).paxosservice(auth 1..24) refresh upgraded, format 0 -> 3 2024-09-17T00:24:16.991 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 bash[56693]: aab3ab73a850570e9284d848e27b6d1efd9bba4063d67a8b5aa8413115306edf 2024-09-17T00:24:16.991 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:16 smithi099 systemd[1]: Started Ceph mon.b for 3aad27a2-7489-11ef-bceb-c7b262605968. 2024-09-17T00:24:16.992 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:24:16 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: ::ffff:172.21.15.99 - - [17/Sep/2024:00:24:16] "GET /metrics HTTP/1.1" 200 38244 "" "Prometheus/2.43.0" 2024-09-17T00:24:17.003 INFO:teuthology.orchestra.run.smithi002.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2024-09-17T00:24:17.003 INFO:teuthology.orchestra.run.smithi002.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-09-17T00:24:17.003 INFO:teuthology.orchestra.run.smithi002.stdout: daemon iscsi.foo.smithi002.vmouem on smithi002 is in unknown state 2024-09-17T00:24:18.967 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:24:18 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[52757]: ts=2024-09-17T00:24:18.659Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd name=CephOSDFlapping index=13 msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.2\", device_class=\"ssd\", hostname=\"smithi002\", instance=\"172.21.15.99:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.2\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"3aad27a2-7489-11ef-bceb-c7b262605968\", cluster_addr=\"172.21.15.2\", device_class=\"ssd\", hostname=\"smithi002\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.2\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-17T00:24:23.759 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:24:23 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[52757]: ts=2024-09-17T00:24:23.505Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes name=CephNodeDiskspaceWarning index=4 msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi002\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi002\", job=\"node\", machine=\"x86_64\", nodename=\"smithi002\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"3aad27a2-7489-11ef-bceb-c7b262605968\", domainname=\"(none)\", instance=\"smithi002\", job=\"node\", machine=\"x86_64\", nodename=\"smithi002\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-17T00:24:25.466 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:25 smithi099 ceph-mon[56712]: 2: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.b 2024-09-17T00:24:25.466 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:25 smithi099 ceph-mon[56712]: fsmap 2024-09-17T00:24:25.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:25 smithi099 ceph-mon[56712]: osdmap e110: 8 total, 8 up, 8 in 2024-09-17T00:24:25.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:25 smithi099 ceph-mon[56712]: mgrmap e40: y(active, since 42s), standbys: x 2024-09-17T00:24:25.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:25 smithi099 ceph-mon[56712]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-09-17T00:24:25.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:25 smithi099 ceph-mon[56712]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-09-17T00:24:25.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:25 smithi099 ceph-mon[56712]: daemon iscsi.foo.smithi002.vmouem on smithi002 is in unknown state 2024-09-17T00:24:25.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:25 smithi099 ceph-mon[56712]: from='client.? ' entity='' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6801/627921530"}]: dispatch 2024-09-17T00:24:25.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:25 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T00:24:25.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:25 smithi099 ceph-mon[56712]: mon.b calling monitor election 2024-09-17T00:24:25.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:25 smithi099 ceph-mon[56712]: mon.c calling monitor election 2024-09-17T00:24:25.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:25 smithi099 ceph-mon[56712]: mon.a calling monitor election 2024-09-17T00:24:25.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:25 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T00:24:25.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:25 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T00:24:25.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:25 smithi099 ceph-mon[56712]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-09-17T00:24:25.468 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:25 smithi099 ceph-mon[56712]: from='client.? ' entity='' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6801/627921530"}]: dispatch 2024-09-17T00:24:25.468 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:25 smithi099 ceph-mon[56712]: monmap epoch 4 2024-09-17T00:24:25.468 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:25 smithi099 ceph-mon[56712]: fsid 3aad27a2-7489-11ef-bceb-c7b262605968 2024-09-17T00:24:25.468 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:25 smithi099 ceph-mon[56712]: last_changed 2024-09-17T00:24:24.140646+0000 2024-09-17T00:24:25.468 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:25 smithi099 ceph-mon[56712]: created 2024-09-17T00:10:42.458448+0000 2024-09-17T00:24:25.468 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:25 smithi099 ceph-mon[56712]: min_mon_release 19 (squid) 2024-09-17T00:24:25.468 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:25 smithi099 ceph-mon[56712]: election_strategy: 1 2024-09-17T00:24:25.468 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:25 smithi099 ceph-mon[56712]: 0: [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] mon.a 2024-09-17T00:24:25.468 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:25 smithi099 ceph-mon[56712]: 1: [v2:172.21.15.2:3301/0,v1:172.21.15.2:6790/0] mon.c 2024-09-17T00:24:25.468 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:25 smithi099 ceph-mon[56712]: 2: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.b 2024-09-17T00:24:25.468 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:25 smithi099 ceph-mon[56712]: fsmap 2024-09-17T00:24:25.468 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:25 smithi099 ceph-mon[56712]: osdmap e110: 8 total, 8 up, 8 in 2024-09-17T00:24:25.468 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:25 smithi099 ceph-mon[56712]: mgrmap e40: y(active, since 42s), standbys: x 2024-09-17T00:24:25.468 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:25 smithi099 ceph-mon[56712]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-09-17T00:24:25.469 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:25 smithi099 ceph-mon[56712]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-09-17T00:24:25.469 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:25 smithi099 ceph-mon[56712]: daemon iscsi.foo.smithi002.vmouem on smithi002 is in unknown state 2024-09-17T00:24:25.469 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:25 smithi099 ceph-mon[56712]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-09-17T00:24:25.480 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:25 smithi002 ceph-mon[63804]: 2: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.b 2024-09-17T00:24:25.480 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:25 smithi002 ceph-mon[63804]: fsmap 2024-09-17T00:24:25.480 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:25 smithi002 ceph-mon[63804]: osdmap e110: 8 total, 8 up, 8 in 2024-09-17T00:24:25.480 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:25 smithi002 ceph-mon[63804]: mgrmap e40: y(active, since 42s), standbys: x 2024-09-17T00:24:25.480 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:25 smithi002 ceph-mon[63804]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-09-17T00:24:25.480 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:25 smithi002 ceph-mon[63804]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-09-17T00:24:25.480 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:25 smithi002 ceph-mon[63804]: daemon iscsi.foo.smithi002.vmouem on smithi002 is in unknown state 2024-09-17T00:24:25.480 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:25 smithi002 ceph-mon[63804]: from='client.? ' entity='' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6801/627921530"}]: dispatch 2024-09-17T00:24:25.480 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:25 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T00:24:25.480 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:25 smithi002 ceph-mon[63804]: mon.b calling monitor election 2024-09-17T00:24:25.481 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:25 smithi002 ceph-mon[63804]: mon.c calling monitor election 2024-09-17T00:24:25.481 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:25 smithi002 ceph-mon[63804]: mon.a calling monitor election 2024-09-17T00:24:25.481 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:25 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T00:24:25.481 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:25 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T00:24:25.481 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:25 smithi002 ceph-mon[63804]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-09-17T00:24:25.481 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:25 smithi002 ceph-mon[63804]: from='client.? ' entity='' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6801/627921530"}]: dispatch 2024-09-17T00:24:25.481 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:25 smithi002 ceph-mon[63804]: monmap epoch 4 2024-09-17T00:24:25.481 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:25 smithi002 ceph-mon[63804]: fsid 3aad27a2-7489-11ef-bceb-c7b262605968 2024-09-17T00:24:25.481 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:25 smithi002 ceph-mon[63804]: last_changed 2024-09-17T00:24:24.140646+0000 2024-09-17T00:24:25.481 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:25 smithi002 ceph-mon[63804]: created 2024-09-17T00:10:42.458448+0000 2024-09-17T00:24:25.481 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:25 smithi002 ceph-mon[63804]: min_mon_release 19 (squid) 2024-09-17T00:24:25.481 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:25 smithi002 ceph-mon[63804]: election_strategy: 1 2024-09-17T00:24:25.481 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:25 smithi002 ceph-mon[63804]: 0: [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] mon.a 2024-09-17T00:24:25.481 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:25 smithi002 ceph-mon[63804]: 1: [v2:172.21.15.2:3301/0,v1:172.21.15.2:6790/0] mon.c 2024-09-17T00:24:25.482 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:25 smithi002 ceph-mon[63804]: 2: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.b 2024-09-17T00:24:25.482 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:25 smithi002 ceph-mon[63804]: fsmap 2024-09-17T00:24:25.482 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:25 smithi002 ceph-mon[63804]: osdmap e110: 8 total, 8 up, 8 in 2024-09-17T00:24:25.482 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:25 smithi002 ceph-mon[63804]: mgrmap e40: y(active, since 42s), standbys: x 2024-09-17T00:24:25.482 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:25 smithi002 ceph-mon[63804]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-09-17T00:24:25.482 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:25 smithi002 ceph-mon[63804]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-09-17T00:24:25.482 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:25 smithi002 ceph-mon[63804]: daemon iscsi.foo.smithi002.vmouem on smithi002 is in unknown state 2024-09-17T00:24:25.482 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:25 smithi002 ceph-mon[63804]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-09-17T00:24:25.483 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:25 smithi002 ceph-mon[66732]: 2: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.b 2024-09-17T00:24:25.483 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:25 smithi002 ceph-mon[66732]: fsmap 2024-09-17T00:24:25.483 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:25 smithi002 ceph-mon[66732]: osdmap e110: 8 total, 8 up, 8 in 2024-09-17T00:24:25.483 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:25 smithi002 ceph-mon[66732]: mgrmap e40: y(active, since 42s), standbys: x 2024-09-17T00:24:25.483 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:25 smithi002 ceph-mon[66732]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-09-17T00:24:25.483 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:25 smithi002 ceph-mon[66732]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-09-17T00:24:25.483 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:25 smithi002 ceph-mon[66732]: daemon iscsi.foo.smithi002.vmouem on smithi002 is in unknown state 2024-09-17T00:24:25.483 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:25 smithi002 ceph-mon[66732]: from='client.? ' entity='' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6801/627921530"}]: dispatch 2024-09-17T00:24:25.483 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:25 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-17T00:24:25.483 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:25 smithi002 ceph-mon[66732]: mon.b calling monitor election 2024-09-17T00:24:25.483 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:25 smithi002 ceph-mon[66732]: mon.c calling monitor election 2024-09-17T00:24:25.483 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:25 smithi002 ceph-mon[66732]: mon.a calling monitor election 2024-09-17T00:24:25.483 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:25 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-17T00:24:25.484 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:25 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-17T00:24:25.484 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:25 smithi002 ceph-mon[66732]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-09-17T00:24:25.484 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:25 smithi002 ceph-mon[66732]: from='client.? ' entity='' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6801/627921530"}]: dispatch 2024-09-17T00:24:25.484 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:25 smithi002 ceph-mon[66732]: monmap epoch 4 2024-09-17T00:24:25.484 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:25 smithi002 ceph-mon[66732]: fsid 3aad27a2-7489-11ef-bceb-c7b262605968 2024-09-17T00:24:25.484 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:25 smithi002 ceph-mon[66732]: last_changed 2024-09-17T00:24:24.140646+0000 2024-09-17T00:24:25.484 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:25 smithi002 ceph-mon[66732]: created 2024-09-17T00:10:42.458448+0000 2024-09-17T00:24:25.484 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:25 smithi002 ceph-mon[66732]: min_mon_release 19 (squid) 2024-09-17T00:24:25.484 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:25 smithi002 ceph-mon[66732]: election_strategy: 1 2024-09-17T00:24:25.484 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:25 smithi002 ceph-mon[66732]: 0: [v2:172.21.15.2:3300/0,v1:172.21.15.2:6789/0] mon.a 2024-09-17T00:24:25.485 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:25 smithi002 ceph-mon[66732]: 1: [v2:172.21.15.2:3301/0,v1:172.21.15.2:6790/0] mon.c 2024-09-17T00:24:25.485 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:25 smithi002 ceph-mon[66732]: 2: [v2:172.21.15.99:3300/0,v1:172.21.15.99:6789/0] mon.b 2024-09-17T00:24:25.485 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:25 smithi002 ceph-mon[66732]: fsmap 2024-09-17T00:24:25.485 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:25 smithi002 ceph-mon[66732]: osdmap e110: 8 total, 8 up, 8 in 2024-09-17T00:24:25.485 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:25 smithi002 ceph-mon[66732]: mgrmap e40: y(active, since 42s), standbys: x 2024-09-17T00:24:25.485 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:25 smithi002 ceph-mon[66732]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-09-17T00:24:25.485 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:25 smithi002 ceph-mon[66732]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-09-17T00:24:25.485 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:25 smithi002 ceph-mon[66732]: daemon iscsi.foo.smithi002.vmouem on smithi002 is in unknown state 2024-09-17T00:24:25.485 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:25 smithi002 ceph-mon[66732]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-09-17T00:24:26.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:26 smithi099 ceph-mon[56712]: from='client.? ' entity='' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6801/627921530"}]': finished 2024-09-17T00:24:26.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:26 smithi099 ceph-mon[56712]: osdmap e111: 8 total, 8 up, 8 in 2024-09-17T00:24:26.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:26 smithi099 ceph-mon[56712]: from='client.? 172.21.15.2:0/1352290948' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/2715952023"}]: dispatch 2024-09-17T00:24:26.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:26 smithi002 ceph-mon[63804]: from='client.? ' entity='' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6801/627921530"}]': finished 2024-09-17T00:24:26.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:26 smithi002 ceph-mon[63804]: osdmap e111: 8 total, 8 up, 8 in 2024-09-17T00:24:26.480 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:26 smithi002 ceph-mon[63804]: from='client.? 172.21.15.2:0/1352290948' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/2715952023"}]: dispatch 2024-09-17T00:24:26.480 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:26 smithi002 ceph-mon[66732]: from='client.? ' entity='' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:6801/627921530"}]': finished 2024-09-17T00:24:26.480 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:26 smithi002 ceph-mon[66732]: osdmap e111: 8 total, 8 up, 8 in 2024-09-17T00:24:26.480 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:26 smithi002 ceph-mon[66732]: from='client.? 172.21.15.2:0/1352290948' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/2715952023"}]: dispatch 2024-09-17T00:24:26.979 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:24:26 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: ::ffff:172.21.15.99 - - [17/Sep/2024:00:24:26] "GET /metrics HTTP/1.1" 200 38244 "" "Prometheus/2.43.0" 2024-09-17T00:24:26.979 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:24:26 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: 2024-09-17T00:24:26.727+0000 7f17efec7640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-09-17T00:24:27.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:27 smithi099 ceph-mon[56712]: from='client.? 172.21.15.2:0/1352290948' entity='client.iscsi.foo.smithi002.vmouem' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/2715952023"}]': finished 2024-09-17T00:24:27.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:27 smithi099 ceph-mon[56712]: osdmap e112: 8 total, 8 up, 8 in 2024-09-17T00:24:27.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:27 smithi099 ceph-mon[56712]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:24:27.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:27 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T00:24:27.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:27 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:27.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:27 smithi002 ceph-mon[63804]: from='client.? 172.21.15.2:0/1352290948' entity='client.iscsi.foo.smithi002.vmouem' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/2715952023"}]': finished 2024-09-17T00:24:27.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:27 smithi002 ceph-mon[63804]: osdmap e112: 8 total, 8 up, 8 in 2024-09-17T00:24:27.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:27 smithi002 ceph-mon[63804]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:24:27.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:27 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T00:24:27.480 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:27 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:27.480 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:27 smithi002 ceph-mon[66732]: from='client.? 172.21.15.2:0/1352290948' entity='client.iscsi.foo.smithi002.vmouem' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.2:0/2715952023"}]': finished 2024-09-17T00:24:27.480 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:27 smithi002 ceph-mon[66732]: osdmap e112: 8 total, 8 up, 8 in 2024-09-17T00:24:27.480 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:27 smithi002 ceph-mon[66732]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:24:27.480 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:27 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T00:24:27.480 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:27 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:28.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:28 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:28.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:28 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:28.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:28 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:28.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:28 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:28.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:28 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:28.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:28 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:28.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:28 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:28.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:28 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:28.480 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:28 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:28.967 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:24:28 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[52757]: ts=2024-09-17T00:24:28.659Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd name=CephOSDFlapping index=13 msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.2\", device_class=\"ssd\", hostname=\"smithi002\", instance=\"172.21.15.99:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.2\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"3aad27a2-7489-11ef-bceb-c7b262605968\", cluster_addr=\"172.21.15.2\", device_class=\"ssd\", hostname=\"smithi002\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.2\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-17T00:24:29.400 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:29 smithi099 ceph-mon[56712]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-17T00:24:29.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:29 smithi002 ceph-mon[66732]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-17T00:24:29.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:29 smithi002 ceph-mon[63804]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-17T00:24:30.807 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:30 smithi099 ceph-mon[56712]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-17T00:24:30.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:30 smithi002 ceph-mon[66732]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-17T00:24:30.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:30 smithi002 ceph-mon[63804]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-17T00:24:32.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:31 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:32.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:31 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:32.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:31 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:24:32.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:31 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:24:32.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:31 smithi099 ceph-mon[56712]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 809 B/s rd, 0 op/s 2024-09-17T00:24:32.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:31 smithi099 ceph-mon[56712]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 896 B/s rd, 1 op/s 2024-09-17T00:24:32.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:31 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:32.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:31 smithi099 ceph-mon[56712]: Reconfiguring mon.a (monmap changed)... 2024-09-17T00:24:32.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:31 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-17T00:24:32.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:31 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-17T00:24:32.218 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:31 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:24:32.218 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:31 smithi099 ceph-mon[56712]: Reconfiguring daemon mon.a on smithi002 2024-09-17T00:24:32.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:31 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:32.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:31 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:32.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:31 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:24:32.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:31 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:24:32.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:31 smithi002 ceph-mon[63804]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 809 B/s rd, 0 op/s 2024-09-17T00:24:32.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:31 smithi002 ceph-mon[63804]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 896 B/s rd, 1 op/s 2024-09-17T00:24:32.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:31 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:32.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:31 smithi002 ceph-mon[63804]: Reconfiguring mon.a (monmap changed)... 2024-09-17T00:24:32.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:31 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-17T00:24:32.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:31 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-17T00:24:32.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:31 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:24:32.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:31 smithi002 ceph-mon[63804]: Reconfiguring daemon mon.a on smithi002 2024-09-17T00:24:32.230 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:31 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:32.230 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:31 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:32.230 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:31 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:24:32.230 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:31 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:24:32.230 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:31 smithi002 ceph-mon[66732]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 809 B/s rd, 0 op/s 2024-09-17T00:24:32.230 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:31 smithi002 ceph-mon[66732]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 896 B/s rd, 1 op/s 2024-09-17T00:24:32.231 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:31 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:32.231 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:31 smithi002 ceph-mon[66732]: Reconfiguring mon.a (monmap changed)... 2024-09-17T00:24:32.231 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:31 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-17T00:24:32.231 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:31 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-17T00:24:32.231 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:31 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:24:32.231 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:31 smithi002 ceph-mon[66732]: Reconfiguring daemon mon.a on smithi002 2024-09-17T00:24:33.717 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:24:33 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[52757]: ts=2024-09-17T00:24:33.505Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes name=CephNodeDiskspaceWarning index=4 msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi002\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi002\", job=\"node\", machine=\"x86_64\", nodename=\"smithi002\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"3aad27a2-7489-11ef-bceb-c7b262605968\", domainname=\"(none)\", instance=\"smithi002\", job=\"node\", machine=\"x86_64\", nodename=\"smithi002\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-17T00:24:33.718 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:33 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:33.718 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:33 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:33.718 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:33 smithi099 ceph-mon[56712]: Reconfiguring mgr.y (monmap changed)... 2024-09-17T00:24:33.718 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:33 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-17T00:24:33.718 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:33 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-17T00:24:33.718 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:33 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-17T00:24:33.718 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:33 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:24:33.719 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:33 smithi099 ceph-mon[56712]: Reconfiguring daemon mgr.y on smithi002 2024-09-17T00:24:33.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:33 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:33.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:33 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:33.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:33 smithi002 ceph-mon[66732]: Reconfiguring mgr.y (monmap changed)... 2024-09-17T00:24:33.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:33 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-17T00:24:33.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:33 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-17T00:24:33.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:33 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-17T00:24:33.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:33 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:24:33.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:33 smithi002 ceph-mon[66732]: Reconfiguring daemon mgr.y on smithi002 2024-09-17T00:24:33.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:33 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:33.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:33 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:33.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:33 smithi002 ceph-mon[63804]: Reconfiguring mgr.y (monmap changed)... 2024-09-17T00:24:33.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:33 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-17T00:24:33.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:33 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-17T00:24:33.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:33 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-17T00:24:33.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:33 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:24:33.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:33 smithi002 ceph-mon[63804]: Reconfiguring daemon mgr.y on smithi002 2024-09-17T00:24:34.702 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:34 smithi002 ceph-mon[66732]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-17T00:24:34.702 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:34 smithi002 ceph-mon[66732]: from='client.44194 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:24:34.702 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:34 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:34.702 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:34 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:34.702 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:34 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-17T00:24:34.702 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:34 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-17T00:24:34.702 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:34 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:24:34.702 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:34 smithi002 ceph-mon[63804]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-17T00:24:34.703 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:34 smithi002 ceph-mon[63804]: from='client.44194 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:24:34.703 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:34 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:34.703 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:34 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:34.703 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:34 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-17T00:24:34.703 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:34 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-17T00:24:34.703 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:34 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:24:34.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:34 smithi099 ceph-mon[56712]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-17T00:24:34.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:34 smithi099 ceph-mon[56712]: from='client.44194 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:24:34.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:34 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:34.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:34 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:34.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:34 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-17T00:24:34.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:34 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-17T00:24:34.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:34 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:24:35.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:35 smithi099 ceph-mon[56712]: Reconfiguring mon.c (monmap changed)... 2024-09-17T00:24:35.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:35 smithi099 ceph-mon[56712]: Reconfiguring daemon mon.c on smithi002 2024-09-17T00:24:35.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:35 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:35.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:35 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:35.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:35 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-09-17T00:24:35.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:35 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:24:35.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:35 smithi002 ceph-mon[66732]: Reconfiguring mon.c (monmap changed)... 2024-09-17T00:24:35.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:35 smithi002 ceph-mon[66732]: Reconfiguring daemon mon.c on smithi002 2024-09-17T00:24:35.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:35 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:35.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:35 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:35.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:35 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-09-17T00:24:35.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:35 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:24:35.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:35 smithi002 ceph-mon[63804]: Reconfiguring mon.c (monmap changed)... 2024-09-17T00:24:35.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:35 smithi002 ceph-mon[63804]: Reconfiguring daemon mon.c on smithi002 2024-09-17T00:24:35.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:35 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:35.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:35 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:35.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:35 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-09-17T00:24:35.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:35 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:24:36.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:36 smithi099 ceph-mon[56712]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:24:36.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:36 smithi099 ceph-mon[56712]: Reconfiguring osd.0 (monmap changed)... 2024-09-17T00:24:36.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:36 smithi099 ceph-mon[56712]: Reconfiguring daemon osd.0 on smithi002 2024-09-17T00:24:36.729 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:24:36 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: ::ffff:172.21.15.99 - - [17/Sep/2024:00:24:36] "GET /metrics HTTP/1.1" 200 38260 "" "Prometheus/2.43.0" 2024-09-17T00:24:36.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:36 smithi002 ceph-mon[66732]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:24:36.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:36 smithi002 ceph-mon[66732]: Reconfiguring osd.0 (monmap changed)... 2024-09-17T00:24:36.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:36 smithi002 ceph-mon[66732]: Reconfiguring daemon osd.0 on smithi002 2024-09-17T00:24:36.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:36 smithi002 ceph-mon[63804]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:24:36.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:36 smithi002 ceph-mon[63804]: Reconfiguring osd.0 (monmap changed)... 2024-09-17T00:24:36.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:36 smithi002 ceph-mon[63804]: Reconfiguring daemon osd.0 on smithi002 2024-09-17T00:24:38.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:37 smithi099 ceph-mon[56712]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 614 B/s rd, 0 op/s 2024-09-17T00:24:38.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:37 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:38.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:37 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:38.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:37 smithi099 ceph-mon[56712]: Reconfiguring osd.1 (monmap changed)... 2024-09-17T00:24:38.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:37 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-09-17T00:24:38.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:37 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:24:38.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:37 smithi099 ceph-mon[56712]: Reconfiguring daemon osd.1 on smithi002 2024-09-17T00:24:38.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:37 smithi002 ceph-mon[63804]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 614 B/s rd, 0 op/s 2024-09-17T00:24:38.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:37 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:38.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:37 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:38.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:37 smithi002 ceph-mon[63804]: Reconfiguring osd.1 (monmap changed)... 2024-09-17T00:24:38.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:37 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-09-17T00:24:38.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:37 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:24:38.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:37 smithi002 ceph-mon[63804]: Reconfiguring daemon osd.1 on smithi002 2024-09-17T00:24:38.230 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:37 smithi002 ceph-mon[66732]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 614 B/s rd, 0 op/s 2024-09-17T00:24:38.230 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:37 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:38.230 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:37 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:38.230 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:37 smithi002 ceph-mon[66732]: Reconfiguring osd.1 (monmap changed)... 2024-09-17T00:24:38.230 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:37 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-09-17T00:24:38.230 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:37 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:24:38.230 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:37 smithi002 ceph-mon[66732]: Reconfiguring daemon osd.1 on smithi002 2024-09-17T00:24:38.967 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:24:38 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[52757]: ts=2024-09-17T00:24:38.659Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd name=CephOSDFlapping index=13 msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.2\", device_class=\"ssd\", hostname=\"smithi002\", instance=\"172.21.15.99:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.2\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"3aad27a2-7489-11ef-bceb-c7b262605968\", cluster_addr=\"172.21.15.2\", device_class=\"ssd\", hostname=\"smithi002\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.2\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-17T00:24:39.710 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:39 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:39.710 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:39 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:39.710 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:39 smithi002 ceph-mon[63804]: Reconfiguring osd.2 (monmap changed)... 2024-09-17T00:24:39.710 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:39 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-09-17T00:24:39.710 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:39 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:24:39.710 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:39 smithi002 ceph-mon[63804]: Reconfiguring daemon osd.2 on smithi002 2024-09-17T00:24:39.710 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:39 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:39.711 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:39 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:39.711 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:39 smithi002 ceph-mon[66732]: Reconfiguring osd.2 (monmap changed)... 2024-09-17T00:24:39.711 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:39 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-09-17T00:24:39.711 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:39 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:24:39.711 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:39 smithi002 ceph-mon[66732]: Reconfiguring daemon osd.2 on smithi002 2024-09-17T00:24:39.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:39 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:39.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:39 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:39.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:39 smithi099 ceph-mon[56712]: Reconfiguring osd.2 (monmap changed)... 2024-09-17T00:24:39.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:39 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-09-17T00:24:39.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:39 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:24:39.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:39 smithi099 ceph-mon[56712]: Reconfiguring daemon osd.2 on smithi002 2024-09-17T00:24:40.408 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:24:40 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[52757]: ts=2024-09-17T00:24:40.086Z caller=alerting.go:366 level=warn component="rule manager" alert="unsupported value type" msg="Expanding alert template failed" err="error executing template __alert_CephMonDown: template: __alert_CephMonDown:1:427: executing \"__alert_CephMonDown\" at : error calling query: found duplicate series for the match group {ceph_daemon=\"mon.a\"} on the right hand-side of the operation: [{ceph_daemon=\"mon.a\", ceph_version=\"ceph version 19.1.1-330-gc2ddf972 (c2ddf9721ef254645201cdbabd9c8f49c30a586f) squid (stable)\", cluster=\"3aad27a2-7489-11ef-bceb-c7b262605968\", hostname=\"smithi002\", instance=\"ceph_cluster\", job=\"ceph\", public_addr=\"172.21.15.2\", rank=\"0\"}, {ceph_daemon=\"mon.a\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", hostname=\"smithi002\", instance=\"172.21.15.99:9283\", job=\"ceph\", public_addr=\"172.21.15.2\", rank=\"0\"}];many-to-many matching not allowed: matching labels must be unique on one side" data="unsupported value type" 2024-09-17T00:24:40.441 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:40 smithi002 ceph-mon[63804]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:24:40.441 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:40 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:40.441 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:40 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:40.441 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:40 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-09-17T00:24:40.441 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:40 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:24:40.442 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:40 smithi002 ceph-mon[66732]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:24:40.442 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:40 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:40.442 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:40 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:40.442 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:40 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-09-17T00:24:40.442 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:40 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:24:40.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:40 smithi099 ceph-mon[56712]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:24:40.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:40 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:40.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:40 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:40.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:40 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-09-17T00:24:40.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:40 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:24:41.675 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:41 smithi002 ceph-mon[66732]: Reconfiguring osd.3 (monmap changed)... 2024-09-17T00:24:41.675 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:41 smithi002 ceph-mon[66732]: Reconfiguring daemon osd.3 on smithi002 2024-09-17T00:24:41.675 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:41 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:41.676 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:41 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:41.676 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:41 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi002.uwdhcs", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T00:24:41.676 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:41 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi002.uwdhcs", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T00:24:41.676 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:41 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:24:41.676 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:41 smithi002 ceph-mon[63804]: Reconfiguring osd.3 (monmap changed)... 2024-09-17T00:24:41.676 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:41 smithi002 ceph-mon[63804]: Reconfiguring daemon osd.3 on smithi002 2024-09-17T00:24:41.676 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:41 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:41.676 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:41 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:41.676 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:41 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi002.uwdhcs", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T00:24:41.676 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:41 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi002.uwdhcs", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T00:24:41.676 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:41 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:24:41.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:41 smithi099 ceph-mon[56712]: Reconfiguring osd.3 (monmap changed)... 2024-09-17T00:24:41.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:41 smithi099 ceph-mon[56712]: Reconfiguring daemon osd.3 on smithi002 2024-09-17T00:24:41.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:41 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:41.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:41 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:41.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:41 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi002.uwdhcs", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T00:24:41.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:41 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi002.uwdhcs", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T00:24:41.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:41 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:24:42.673 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:42 smithi002 ceph-mon[66732]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-17T00:24:42.673 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:42 smithi002 ceph-mon[66732]: Reconfiguring rgw.foo.smithi002.uwdhcs (monmap changed)... 2024-09-17T00:24:42.673 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:42 smithi002 ceph-mon[66732]: Reconfiguring daemon rgw.foo.smithi002.uwdhcs on smithi002 2024-09-17T00:24:42.673 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:42 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T00:24:42.674 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:42 smithi002 ceph-mon[63804]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-17T00:24:42.674 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:42 smithi002 ceph-mon[63804]: Reconfiguring rgw.foo.smithi002.uwdhcs (monmap changed)... 2024-09-17T00:24:42.674 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:42 smithi002 ceph-mon[63804]: Reconfiguring daemon rgw.foo.smithi002.uwdhcs on smithi002 2024-09-17T00:24:42.674 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:42 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T00:24:42.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:42 smithi099 ceph-mon[56712]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-17T00:24:42.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:42 smithi099 ceph-mon[56712]: Reconfiguring rgw.foo.smithi002.uwdhcs (monmap changed)... 2024-09-17T00:24:42.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:42 smithi099 ceph-mon[56712]: Reconfiguring daemon rgw.foo.smithi002.uwdhcs on smithi002 2024-09-17T00:24:42.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:42 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T00:24:43.967 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:24:43 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[52757]: ts=2024-09-17T00:24:43.506Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes name=CephNodeDiskspaceWarning index=4 msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi002\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi002\", job=\"node\", machine=\"x86_64\", nodename=\"smithi002\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"3aad27a2-7489-11ef-bceb-c7b262605968\", domainname=\"(none)\", instance=\"smithi002\", job=\"node\", machine=\"x86_64\", nodename=\"smithi002\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-17T00:24:43.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:43 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:43.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:43 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:43.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:43 smithi099 ceph-mon[56712]: Reconfiguring rgw.smpl.smithi002.rqnxii (monmap changed)... 2024-09-17T00:24:43.968 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:43 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi002.rqnxii", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T00:24:43.968 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:43 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi002.rqnxii", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T00:24:43.968 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:43 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:24:43.968 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:43 smithi099 ceph-mon[56712]: Reconfiguring daemon rgw.smpl.smithi002.rqnxii on smithi002 2024-09-17T00:24:43.968 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:43 smithi099 ceph-mon[56712]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:24:43.968 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:43 smithi099 ceph-mon[56712]: from='client.44194 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:24:44.164 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:43 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:44.164 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:43 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:44.164 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:43 smithi002 ceph-mon[66732]: Reconfiguring rgw.smpl.smithi002.rqnxii (monmap changed)... 2024-09-17T00:24:44.164 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:43 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi002.rqnxii", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T00:24:44.164 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:43 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi002.rqnxii", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T00:24:44.164 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:43 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:24:44.164 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:43 smithi002 ceph-mon[66732]: Reconfiguring daemon rgw.smpl.smithi002.rqnxii on smithi002 2024-09-17T00:24:44.164 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:43 smithi002 ceph-mon[66732]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:24:44.164 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:43 smithi002 ceph-mon[66732]: from='client.44194 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:24:44.165 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:43 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:44.165 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:43 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:44.165 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:43 smithi002 ceph-mon[63804]: Reconfiguring rgw.smpl.smithi002.rqnxii (monmap changed)... 2024-09-17T00:24:44.165 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:43 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi002.rqnxii", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T00:24:44.165 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:43 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi002.rqnxii", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T00:24:44.165 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:43 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:24:44.165 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:43 smithi002 ceph-mon[63804]: Reconfiguring daemon rgw.smpl.smithi002.rqnxii on smithi002 2024-09-17T00:24:44.165 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:43 smithi002 ceph-mon[63804]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:24:44.165 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:43 smithi002 ceph-mon[63804]: from='client.44194 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:24:45.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:45 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:45.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:45 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:45.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:45 smithi002 ceph-mon[63804]: Reconfiguring iscsi.foo.smithi002.vmouem (dependencies changed)... 2024-09-17T00:24:45.480 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:45 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi002.vmouem", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-17T00:24:45.480 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:45 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi002.vmouem", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-17T00:24:45.480 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:45 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:24:45.480 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:45 smithi002 ceph-mon[63804]: Reconfiguring daemon iscsi.foo.smithi002.vmouem on smithi002 2024-09-17T00:24:45.480 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:45 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:45.480 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:45 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:45.480 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:45 smithi002 ceph-mon[66732]: Reconfiguring iscsi.foo.smithi002.vmouem (dependencies changed)... 2024-09-17T00:24:45.480 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:45 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi002.vmouem", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-17T00:24:45.480 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:45 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi002.vmouem", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-17T00:24:45.480 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:45 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:24:45.481 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:45 smithi002 ceph-mon[66732]: Reconfiguring daemon iscsi.foo.smithi002.vmouem on smithi002 2024-09-17T00:24:45.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:45 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:45.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:45 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:45.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:45 smithi099 ceph-mon[56712]: Reconfiguring iscsi.foo.smithi002.vmouem (dependencies changed)... 2024-09-17T00:24:45.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:45 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi002.vmouem", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-17T00:24:45.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:45 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi002.vmouem", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-17T00:24:45.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:45 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:24:45.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:45 smithi099 ceph-mon[56712]: Reconfiguring daemon iscsi.foo.smithi002.vmouem on smithi002 2024-09-17T00:24:46.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:46 smithi002 ceph-mon[63804]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:24:46.480 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:46 smithi002 ceph-mon[66732]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:24:46.721 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:46 smithi099 ceph-mon[56712]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:24:46.952 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:24:46 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: ::ffff:172.21.15.99 - - [17/Sep/2024:00:24:46] "GET /metrics HTTP/1.1" 200 38257 "" "Prometheus/2.43.0" 2024-09-17T00:24:47.394 INFO:teuthology.orchestra.run.smithi002.stdout:true 2024-09-17T00:24:47.979 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:24:47 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: 2024-09-17T00:24:47.557+0000 7f17e1e2b640 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi002.vmouem ... 2024-09-17T00:24:47.980 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:24:47 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: Creating ceph-iscsi config... 2024-09-17T00:24:47.980 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:24:47 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/iscsi-gateway.cfg 2024-09-17T00:24:47.980 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:24:47 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/tcmu-runner-entrypoint.sh 2024-09-17T00:24:47.980 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:24:47 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: Non-zero exit code 1 from systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem 2024-09-17T00:24:47.980 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:24:47 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: systemctl: stderr Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:24:47.981 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:24:47 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: systemctl: stderr See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:24:47.981 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:24:47 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: Traceback (most recent call last): 2024-09-17T00:24:47.981 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:24:47 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-17T00:24:47.981 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:24:47 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: return _run_code(code, main_globals, None, 2024-09-17T00:24:47.981 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:24:47 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-17T00:24:47.981 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:24:47 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: exec(code, run_globals) 2024-09-17T00:24:47.981 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:24:47 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5579, in 2024-09-17T00:24:47.981 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:24:47 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5567, in main 2024-09-17T00:24:47.981 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:24:47 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3049, in command_deploy_from 2024-09-17T00:24:47.981 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:24:47 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3084, in _common_deploy 2024-09-17T00:24:47.981 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:24:47 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3104, in _deploy_daemon_container 2024-09-17T00:24:47.981 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:24:47 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 1124, in deploy_daemon 2024-09-17T00:24:47.981 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:24:47 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/tmp/tmpnroy8d0w.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-17T00:24:47.981 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:24:47 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: RuntimeError: Failed command: systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem: Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:24:47.982 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:24:47 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:24:47.982 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:24:47 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: Traceback (most recent call last): 2024-09-17T00:24:47.982 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:24:47 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1134, in _check_daemons 2024-09-17T00:24:47.982 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:24:47 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: self.mgr._daemon_action(daemon_spec, action=action) 2024-09-17T00:24:47.982 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:24:47 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2473, in _daemon_action 2024-09-17T00:24:47.982 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:24:47 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: return self.wait_async( 2024-09-17T00:24:47.982 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:24:47 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/usr/share/ceph/mgr/cephadm/module.py", line 796, in wait_async 2024-09-17T00:24:47.983 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:24:47 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: return self.event_loop.get_result(coro, timeout) 2024-09-17T00:24:47.983 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:24:47 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-09-17T00:24:47.983 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:24:47 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: return future.result(timeout) 2024-09-17T00:24:47.983 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:24:47 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/lib64/python3.9/concurrent/futures/_base.py", line 446, in result 2024-09-17T00:24:47.983 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:24:47 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: return self.__get_result() 2024-09-17T00:24:47.983 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:24:47 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/lib64/python3.9/concurrent/futures/_base.py", line 391, in __get_result 2024-09-17T00:24:47.983 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:24:47 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: raise self._exception 2024-09-17T00:24:47.983 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:24:47 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1370, in _create_daemon 2024-09-17T00:24:47.983 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:24:47 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: out, err, code = await self._run_cephadm( 2024-09-17T00:24:47.983 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:24:47 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1713, in _run_cephadm 2024-09-17T00:24:47.983 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:24:47 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: raise OrchestratorError( 2024-09-17T00:24:47.983 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:24:47 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi002.vmouem ... 2024-09-17T00:24:47.983 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:24:47 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: Creating ceph-iscsi config... 2024-09-17T00:24:47.983 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:24:47 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/iscsi-gateway.cfg 2024-09-17T00:24:47.984 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:24:47 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/tcmu-runner-entrypoint.sh 2024-09-17T00:24:47.984 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:24:47 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: Non-zero exit code 1 from systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem 2024-09-17T00:24:47.984 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:24:47 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: systemctl: stderr Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:24:47.984 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:24:47 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: systemctl: stderr See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:24:47.984 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:24:47 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: Traceback (most recent call last): 2024-09-17T00:24:47.984 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:24:47 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-17T00:24:47.984 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:24:47 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: return _run_code(code, main_globals, None, 2024-09-17T00:24:47.984 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:24:47 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-17T00:24:47.984 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:24:47 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: exec(code, run_globals) 2024-09-17T00:24:47.984 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:24:47 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5579, in 2024-09-17T00:24:47.984 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:24:47 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5567, in main 2024-09-17T00:24:47.984 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:24:47 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3049, in command_deploy_from 2024-09-17T00:24:47.984 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:24:47 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3084, in _common_deploy 2024-09-17T00:24:47.985 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:24:47 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3104, in _deploy_daemon_container 2024-09-17T00:24:47.985 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:24:47 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 1124, in deploy_daemon 2024-09-17T00:24:47.985 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:24:47 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/tmp/tmpnroy8d0w.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-17T00:24:47.985 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:24:47 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: RuntimeError: Failed command: systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem: Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:24:47.985 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:24:47 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:24:48.082 INFO:teuthology.orchestra.run.smithi002.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-17T00:24:48.082 INFO:teuthology.orchestra.run.smithi002.stdout:alertmanager.a smithi002 *:9093,9094 running (2m) 43s ago 7m 18.8M - 0.25.0 c8568f914cd2 a21e13b3d677 2024-09-17T00:24:48.082 INFO:teuthology.orchestra.run.smithi002.stdout:grafana.a smithi099 *:3000 running (2m) 20s ago 7m 46.9M - 8.3.5 dad864ee21e9 bafb824a3c08 2024-09-17T00:24:48.083 INFO:teuthology.orchestra.run.smithi002.stdout:iscsi.foo.smithi002.vmouem smithi002 unknown 43s ago 6m - - 2024-09-17T00:24:48.083 INFO:teuthology.orchestra.run.smithi002.stdout:mgr.x smithi099 *:8443,9283,8765 running (91s) 20s ago 12m 439M - 19.1.1-330-gc2ddf972 ebb0615149de 6aa2ccef9ca0 2024-09-17T00:24:48.083 INFO:teuthology.orchestra.run.smithi002.stdout:mgr.y smithi002 *:8443,9283,8765 running (2m) 43s ago 13m 492M - 19.1.1-330-gc2ddf972 ebb0615149de 8838897e8449 2024-09-17T00:24:48.083 INFO:teuthology.orchestra.run.smithi002.stdout:mon.a smithi002 running (75s) 43s ago 13m 39.9M 2048M 19.1.1-330-gc2ddf972 ebb0615149de a035def53ef6 2024-09-17T00:24:48.083 INFO:teuthology.orchestra.run.smithi002.stdout:mon.b smithi099 running (31s) 20s ago 12m 17.5M 2048M 19.1.1-330-gc2ddf972 ebb0615149de aab3ab73a850 2024-09-17T00:24:48.083 INFO:teuthology.orchestra.run.smithi002.stdout:mon.c smithi002 running (49s) 43s ago 12m 18.5M 2048M 19.1.1-330-gc2ddf972 ebb0615149de 8a510255ae66 2024-09-17T00:24:48.084 INFO:teuthology.orchestra.run.smithi002.stdout:node-exporter.a smithi002 *:9100 running (2m) 43s ago 8m 15.0M - 1.5.0 0da6a335fe13 1030ae01bb32 2024-09-17T00:24:48.084 INFO:teuthology.orchestra.run.smithi002.stdout:node-exporter.b smithi099 *:9100 running (2m) 20s ago 8m 16.8M - 1.5.0 0da6a335fe13 01fcd35dca8a 2024-09-17T00:24:48.084 INFO:teuthology.orchestra.run.smithi002.stdout:osd.0 smithi002 running (11m) 43s ago 11m 53.6M 2737M 17.2.0 e1d6a67b021e a0ad0fb7aa34 2024-09-17T00:24:48.084 INFO:teuthology.orchestra.run.smithi002.stdout:osd.1 smithi002 running (11m) 43s ago 11m 56.0M 2737M 17.2.0 e1d6a67b021e 7cb62c311c7d 2024-09-17T00:24:48.084 INFO:teuthology.orchestra.run.smithi002.stdout:osd.2 smithi002 running (11m) 43s ago 11m 52.5M 2737M 17.2.0 e1d6a67b021e eafddacb955e 2024-09-17T00:24:48.084 INFO:teuthology.orchestra.run.smithi002.stdout:osd.3 smithi002 running (10m) 43s ago 10m 55.4M 2737M 17.2.0 e1d6a67b021e fd99d3764412 2024-09-17T00:24:48.085 INFO:teuthology.orchestra.run.smithi002.stdout:osd.4 smithi099 running (10m) 20s ago 10m 55.4M 2992M 17.2.0 e1d6a67b021e 0af1827b658d 2024-09-17T00:24:48.085 INFO:teuthology.orchestra.run.smithi002.stdout:osd.5 smithi099 running (9m) 20s ago 9m 54.2M 2992M 17.2.0 e1d6a67b021e 4f7ef3aefc58 2024-09-17T00:24:48.085 INFO:teuthology.orchestra.run.smithi002.stdout:osd.6 smithi099 running (9m) 20s ago 9m 51.0M 2992M 17.2.0 e1d6a67b021e 3213f5e6952a 2024-09-17T00:24:48.085 INFO:teuthology.orchestra.run.smithi002.stdout:osd.7 smithi099 running (9m) 20s ago 9m 56.6M 2992M 17.2.0 e1d6a67b021e 63be3a1ed0a1 2024-09-17T00:24:48.085 INFO:teuthology.orchestra.run.smithi002.stdout:prometheus.a smithi099 *:9095 running (96s) 20s ago 8m 50.1M - 2.43.0 a07b618ecd1d bf32c2ae11dc 2024-09-17T00:24:48.085 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.foo.smithi002.uwdhcs smithi002 *:8000 running (6m) 43s ago 6m 75.3M - 17.2.0 e1d6a67b021e 9c35813f459a 2024-09-17T00:24:48.086 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.foo.smithi099.utzfap smithi099 *:8000 running (6m) 20s ago 6m 76.0M - 17.2.0 e1d6a67b021e 34ec989160a6 2024-09-17T00:24:48.086 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.smpl.smithi002.rqnxii smithi002 *:80 running (6m) 43s ago 6m 73.6M - 17.2.0 e1d6a67b021e 0e180d0afb99 2024-09-17T00:24:48.086 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.smpl.smithi099.nzvkpm smithi099 *:80 running (6m) 20s ago 6m 76.5M - 17.2.0 e1d6a67b021e 6a068e82ac72 2024-09-17T00:24:48.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:48 smithi002 ceph-mon[66732]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:24:48.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:48 smithi002 ceph-mon[66732]: from='client.44221 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:24:48.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:48 smithi002 ceph-mon[66732]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi002.vmouem ... 2024-09-17T00:24:48.479 INFO:journalctl@ceph.mon.c.smithi002.stdout: Creating ceph-iscsi config... 2024-09-17T00:24:48.480 INFO:journalctl@ceph.mon.c.smithi002.stdout: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/iscsi-gateway.cfg 2024-09-17T00:24:48.480 INFO:journalctl@ceph.mon.c.smithi002.stdout: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/tcmu-runner-entrypoint.sh 2024-09-17T00:24:48.480 INFO:journalctl@ceph.mon.c.smithi002.stdout: Non-zero exit code 1 from systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem 2024-09-17T00:24:48.480 INFO:journalctl@ceph.mon.c.smithi002.stdout: systemctl: stderr Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:24:48.480 INFO:journalctl@ceph.mon.c.smithi002.stdout: systemctl: stderr See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:24:48.480 INFO:journalctl@ceph.mon.c.smithi002.stdout: Traceback (most recent call last): 2024-09-17T00:24:48.480 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-17T00:24:48.480 INFO:journalctl@ceph.mon.c.smithi002.stdout: return _run_code(code, main_globals, None, 2024-09-17T00:24:48.480 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-17T00:24:48.480 INFO:journalctl@ceph.mon.c.smithi002.stdout: exec(code, run_globals) 2024-09-17T00:24:48.480 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5579, in 2024-09-17T00:24:48.480 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5567, in main 2024-09-17T00:24:48.480 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3049, in command_deploy_from 2024-09-17T00:24:48.480 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3084, in _common_deploy 2024-09-17T00:24:48.481 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3104, in _deploy_daemon_container 2024-09-17T00:24:48.481 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 1124, in deploy_daemon 2024-09-17T00:24:48.481 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-17T00:24:48.481 INFO:journalctl@ceph.mon.c.smithi002.stdout: RuntimeError: Failed command: systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem: Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:24:48.481 INFO:journalctl@ceph.mon.c.smithi002.stdout: See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:24:48.481 INFO:journalctl@ceph.mon.c.smithi002.stdout: Traceback (most recent call last): 2024-09-17T00:24:48.481 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1134, in _check_daemons 2024-09-17T00:24:48.481 INFO:journalctl@ceph.mon.c.smithi002.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-09-17T00:24:48.481 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2473, in _daemon_action 2024-09-17T00:24:48.481 INFO:journalctl@ceph.mon.c.smithi002.stdout: return self.wait_async( 2024-09-17T00:24:48.481 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 796, in wait_async 2024-09-17T00:24:48.481 INFO:journalctl@ceph.mon.c.smithi002.stdout: return self.event_loop.get_result(coro, timeout) 2024-09-17T00:24:48.481 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-09-17T00:24:48.482 INFO:journalctl@ceph.mon.c.smithi002.stdout: return future.result(timeout) 2024-09-17T00:24:48.482 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/lib64/python3.9/concurrent/futures/_base.py", line 446, in result 2024-09-17T00:24:48.482 INFO:journalctl@ceph.mon.c.smithi002.stdout: return self.__get_result() 2024-09-17T00:24:48.482 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/lib64/python3.9/concurrent/futures/_base.py", line 391, in __get_result 2024-09-17T00:24:48.482 INFO:journalctl@ceph.mon.c.smithi002.stdout: raise self._exception 2024-09-17T00:24:48.482 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1370, in _create_daemon 2024-09-17T00:24:48.482 INFO:journalctl@ceph.mon.c.smithi002.stdout: out, err, code = await self._run_cephadm( 2024-09-17T00:24:48.482 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1713, in _run_cephadm 2024-09-17T00:24:48.482 INFO:journalctl@ceph.mon.c.smithi002.stdout: raise OrchestratorError( 2024-09-17T00:24:48.482 INFO:journalctl@ceph.mon.c.smithi002.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi002.vmouem ... 2024-09-17T00:24:48.482 INFO:journalctl@ceph.mon.c.smithi002.stdout: Creating ceph-iscsi config... 2024-09-17T00:24:48.482 INFO:journalctl@ceph.mon.c.smithi002.stdout: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/iscsi-gateway.cfg 2024-09-17T00:24:48.482 INFO:journalctl@ceph.mon.c.smithi002.stdout: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/tcmu-runner-entrypoint.sh 2024-09-17T00:24:48.482 INFO:journalctl@ceph.mon.c.smithi002.stdout: Non-zero exit code 1 from systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem 2024-09-17T00:24:48.483 INFO:journalctl@ceph.mon.c.smithi002.stdout: systemctl: stderr Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:24:48.483 INFO:journalctl@ceph.mon.c.smithi002.stdout: systemctl: stderr See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:24:48.483 INFO:journalctl@ceph.mon.c.smithi002.stdout: Traceback (most recent call last): 2024-09-17T00:24:48.483 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-17T00:24:48.483 INFO:journalctl@ceph.mon.c.smithi002.stdout: return _run_code(code, main_globals, None, 2024-09-17T00:24:48.483 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-17T00:24:48.483 INFO:journalctl@ceph.mon.c.smithi002.stdout: exec(code, run_globals) 2024-09-17T00:24:48.483 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5579, in 2024-09-17T00:24:48.483 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5567, in main 2024-09-17T00:24:48.483 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3049, in command_deploy_from 2024-09-17T00:24:48.483 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3084, in _common_deploy 2024-09-17T00:24:48.483 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3104, in _deploy_daemon_container 2024-09-17T00:24:48.483 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 1124, in deploy_daemon 2024-09-17T00:24:48.483 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-17T00:24:48.483 INFO:journalctl@ceph.mon.c.smithi002.stdout: RuntimeError: Failed command: systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem: Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:24:48.484 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:48 smithi002 ceph-mon[63804]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:24:48.484 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:48 smithi002 ceph-mon[63804]: from='client.44221 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:24:48.484 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:48 smithi002 ceph-mon[63804]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi002.vmouem ... 2024-09-17T00:24:48.484 INFO:journalctl@ceph.mon.a.smithi002.stdout: Creating ceph-iscsi config... 2024-09-17T00:24:48.484 INFO:journalctl@ceph.mon.a.smithi002.stdout: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/iscsi-gateway.cfg 2024-09-17T00:24:48.484 INFO:journalctl@ceph.mon.a.smithi002.stdout: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/tcmu-runner-entrypoint.sh 2024-09-17T00:24:48.484 INFO:journalctl@ceph.mon.a.smithi002.stdout: Non-zero exit code 1 from systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem 2024-09-17T00:24:48.484 INFO:journalctl@ceph.mon.a.smithi002.stdout: systemctl: stderr Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:24:48.484 INFO:journalctl@ceph.mon.a.smithi002.stdout: systemctl: stderr See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:24:48.485 INFO:journalctl@ceph.mon.a.smithi002.stdout: Traceback (most recent call last): 2024-09-17T00:24:48.485 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-17T00:24:48.485 INFO:journalctl@ceph.mon.a.smithi002.stdout: return _run_code(code, main_globals, None, 2024-09-17T00:24:48.485 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-17T00:24:48.485 INFO:journalctl@ceph.mon.a.smithi002.stdout: exec(code, run_globals) 2024-09-17T00:24:48.485 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5579, in 2024-09-17T00:24:48.485 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5567, in main 2024-09-17T00:24:48.485 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3049, in command_deploy_from 2024-09-17T00:24:48.485 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3084, in _common_deploy 2024-09-17T00:24:48.485 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3104, in _deploy_daemon_container 2024-09-17T00:24:48.485 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 1124, in deploy_daemon 2024-09-17T00:24:48.485 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-17T00:24:48.486 INFO:journalctl@ceph.mon.a.smithi002.stdout: RuntimeError: Failed command: systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem: Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:24:48.486 INFO:journalctl@ceph.mon.a.smithi002.stdout: See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:24:48.486 INFO:journalctl@ceph.mon.a.smithi002.stdout: Traceback (most recent call last): 2024-09-17T00:24:48.486 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1134, in _check_daemons 2024-09-17T00:24:48.486 INFO:journalctl@ceph.mon.a.smithi002.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-09-17T00:24:48.486 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2473, in _daemon_action 2024-09-17T00:24:48.486 INFO:journalctl@ceph.mon.a.smithi002.stdout: return self.wait_async( 2024-09-17T00:24:48.486 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 796, in wait_async 2024-09-17T00:24:48.486 INFO:journalctl@ceph.mon.a.smithi002.stdout: return self.event_loop.get_result(coro, timeout) 2024-09-17T00:24:48.486 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-09-17T00:24:48.486 INFO:journalctl@ceph.mon.a.smithi002.stdout: return future.result(timeout) 2024-09-17T00:24:48.486 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/lib64/python3.9/concurrent/futures/_base.py", line 446, in result 2024-09-17T00:24:48.486 INFO:journalctl@ceph.mon.a.smithi002.stdout: return self.__get_result() 2024-09-17T00:24:48.486 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/lib64/python3.9/concurrent/futures/_base.py", line 391, in __get_result 2024-09-17T00:24:48.487 INFO:journalctl@ceph.mon.a.smithi002.stdout: raise self._exception 2024-09-17T00:24:48.487 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1370, in _create_daemon 2024-09-17T00:24:48.487 INFO:journalctl@ceph.mon.a.smithi002.stdout: out, err, code = await self._run_cephadm( 2024-09-17T00:24:48.487 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1713, in _run_cephadm 2024-09-17T00:24:48.487 INFO:journalctl@ceph.mon.a.smithi002.stdout: raise OrchestratorError( 2024-09-17T00:24:48.487 INFO:journalctl@ceph.mon.a.smithi002.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi002.vmouem ... 2024-09-17T00:24:48.487 INFO:journalctl@ceph.mon.a.smithi002.stdout: Creating ceph-iscsi config... 2024-09-17T00:24:48.487 INFO:journalctl@ceph.mon.a.smithi002.stdout: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/iscsi-gateway.cfg 2024-09-17T00:24:48.487 INFO:journalctl@ceph.mon.a.smithi002.stdout: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/tcmu-runner-entrypoint.sh 2024-09-17T00:24:48.487 INFO:journalctl@ceph.mon.a.smithi002.stdout: Non-zero exit code 1 from systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem 2024-09-17T00:24:48.487 INFO:journalctl@ceph.mon.a.smithi002.stdout: systemctl: stderr Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:24:48.487 INFO:journalctl@ceph.mon.a.smithi002.stdout: systemctl: stderr See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:24:48.487 INFO:journalctl@ceph.mon.a.smithi002.stdout: Traceback (most recent call last): 2024-09-17T00:24:48.488 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-17T00:24:48.488 INFO:journalctl@ceph.mon.a.smithi002.stdout: return _run_code(code, main_globals, None, 2024-09-17T00:24:48.488 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-17T00:24:48.488 INFO:journalctl@ceph.mon.a.smithi002.stdout: exec(code, run_globals) 2024-09-17T00:24:48.488 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5579, in 2024-09-17T00:24:48.488 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5567, in main 2024-09-17T00:24:48.488 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3049, in command_deploy_from 2024-09-17T00:24:48.488 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3084, in _common_deploy 2024-09-17T00:24:48.488 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3104, in _deploy_daemon_container 2024-09-17T00:24:48.489 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 1124, in deploy_daemon 2024-09-17T00:24:48.489 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-17T00:24:48.489 INFO:journalctl@ceph.mon.a.smithi002.stdout: RuntimeError: Failed command: systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem: Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:24:48.489 INFO:journalctl@ceph.mon.c.smithi002.stdout: See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:24:48.489 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:48 smithi002 ceph-mon[66732]: Reconfiguring mon.b (monmap changed)... 2024-09-17T00:24:48.489 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:48 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-17T00:24:48.489 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:48 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-17T00:24:48.489 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:48 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:24:48.489 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:48 smithi002 ceph-mon[66732]: Reconfiguring daemon mon.b on smithi099 2024-09-17T00:24:48.490 INFO:journalctl@ceph.mon.a.smithi002.stdout: See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:24:48.490 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:48 smithi002 ceph-mon[63804]: Reconfiguring mon.b (monmap changed)... 2024-09-17T00:24:48.490 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:48 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-17T00:24:48.490 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:48 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-17T00:24:48.490 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:48 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:24:48.490 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:48 smithi002 ceph-mon[63804]: Reconfiguring daemon mon.b on smithi099 2024-09-17T00:24:48.503 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-09-17T00:24:48.503 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": { 2024-09-17T00:24:48.503 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 19.1.1-330-gc2ddf972 (c2ddf9721ef254645201cdbabd9c8f49c30a586f) squid (stable)": 3 2024-09-17T00:24:48.503 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-09-17T00:24:48.503 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": { 2024-09-17T00:24:48.503 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 19.1.1-330-gc2ddf972 (c2ddf9721ef254645201cdbabd9c8f49c30a586f) squid (stable)": 2 2024-09-17T00:24:48.503 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-09-17T00:24:48.503 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": { 2024-09-17T00:24:48.503 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-09-17T00:24:48.503 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-09-17T00:24:48.504 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw": { 2024-09-17T00:24:48.504 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-09-17T00:24:48.504 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-09-17T00:24:48.504 INFO:teuthology.orchestra.run.smithi002.stdout: "overall": { 2024-09-17T00:24:48.504 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 12, 2024-09-17T00:24:48.504 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 19.1.1-330-gc2ddf972 (c2ddf9721ef254645201cdbabd9c8f49c30a586f) squid (stable)": 5 2024-09-17T00:24:48.504 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-09-17T00:24:48.504 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-09-17T00:24:48.657 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:48 smithi099 ceph-mon[56712]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:24:48.658 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:48 smithi099 ceph-mon[56712]: from='client.44221 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:24:48.658 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:48 smithi099 ceph-mon[56712]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi002.vmouem ... 2024-09-17T00:24:48.658 INFO:journalctl@ceph.mon.b.smithi099.stdout: Creating ceph-iscsi config... 2024-09-17T00:24:48.658 INFO:journalctl@ceph.mon.b.smithi099.stdout: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/iscsi-gateway.cfg 2024-09-17T00:24:48.658 INFO:journalctl@ceph.mon.b.smithi099.stdout: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/tcmu-runner-entrypoint.sh 2024-09-17T00:24:48.658 INFO:journalctl@ceph.mon.b.smithi099.stdout: Non-zero exit code 1 from systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem 2024-09-17T00:24:48.658 INFO:journalctl@ceph.mon.b.smithi099.stdout: systemctl: stderr Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:24:48.659 INFO:journalctl@ceph.mon.b.smithi099.stdout: systemctl: stderr See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:24:48.659 INFO:journalctl@ceph.mon.b.smithi099.stdout: Traceback (most recent call last): 2024-09-17T00:24:48.659 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-17T00:24:48.659 INFO:journalctl@ceph.mon.b.smithi099.stdout: return _run_code(code, main_globals, None, 2024-09-17T00:24:48.659 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-17T00:24:48.659 INFO:journalctl@ceph.mon.b.smithi099.stdout: exec(code, run_globals) 2024-09-17T00:24:48.659 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5579, in 2024-09-17T00:24:48.659 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5567, in main 2024-09-17T00:24:48.660 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3049, in command_deploy_from 2024-09-17T00:24:48.660 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3084, in _common_deploy 2024-09-17T00:24:48.660 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3104, in _deploy_daemon_container 2024-09-17T00:24:48.660 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 1124, in deploy_daemon 2024-09-17T00:24:48.660 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-17T00:24:48.660 INFO:journalctl@ceph.mon.b.smithi099.stdout: RuntimeError: Failed command: systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem: Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:24:48.660 INFO:journalctl@ceph.mon.b.smithi099.stdout: See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:24:48.661 INFO:journalctl@ceph.mon.b.smithi099.stdout: Traceback (most recent call last): 2024-09-17T00:24:48.661 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1134, in _check_daemons 2024-09-17T00:24:48.661 INFO:journalctl@ceph.mon.b.smithi099.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-09-17T00:24:48.661 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2473, in _daemon_action 2024-09-17T00:24:48.661 INFO:journalctl@ceph.mon.b.smithi099.stdout: return self.wait_async( 2024-09-17T00:24:48.661 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 796, in wait_async 2024-09-17T00:24:48.662 INFO:journalctl@ceph.mon.b.smithi099.stdout: return self.event_loop.get_result(coro, timeout) 2024-09-17T00:24:48.662 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-09-17T00:24:48.662 INFO:journalctl@ceph.mon.b.smithi099.stdout: return future.result(timeout) 2024-09-17T00:24:48.662 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/lib64/python3.9/concurrent/futures/_base.py", line 446, in result 2024-09-17T00:24:48.662 INFO:journalctl@ceph.mon.b.smithi099.stdout: return self.__get_result() 2024-09-17T00:24:48.662 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/lib64/python3.9/concurrent/futures/_base.py", line 391, in __get_result 2024-09-17T00:24:48.662 INFO:journalctl@ceph.mon.b.smithi099.stdout: raise self._exception 2024-09-17T00:24:48.662 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1370, in _create_daemon 2024-09-17T00:24:48.662 INFO:journalctl@ceph.mon.b.smithi099.stdout: out, err, code = await self._run_cephadm( 2024-09-17T00:24:48.663 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1713, in _run_cephadm 2024-09-17T00:24:48.663 INFO:journalctl@ceph.mon.b.smithi099.stdout: raise OrchestratorError( 2024-09-17T00:24:48.663 INFO:journalctl@ceph.mon.b.smithi099.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi002.vmouem ... 2024-09-17T00:24:48.663 INFO:journalctl@ceph.mon.b.smithi099.stdout: Creating ceph-iscsi config... 2024-09-17T00:24:48.663 INFO:journalctl@ceph.mon.b.smithi099.stdout: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/iscsi-gateway.cfg 2024-09-17T00:24:48.663 INFO:journalctl@ceph.mon.b.smithi099.stdout: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/tcmu-runner-entrypoint.sh 2024-09-17T00:24:48.663 INFO:journalctl@ceph.mon.b.smithi099.stdout: Non-zero exit code 1 from systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem 2024-09-17T00:24:48.663 INFO:journalctl@ceph.mon.b.smithi099.stdout: systemctl: stderr Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:24:48.664 INFO:journalctl@ceph.mon.b.smithi099.stdout: systemctl: stderr See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:24:48.664 INFO:journalctl@ceph.mon.b.smithi099.stdout: Traceback (most recent call last): 2024-09-17T00:24:48.664 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-17T00:24:48.664 INFO:journalctl@ceph.mon.b.smithi099.stdout: return _run_code(code, main_globals, None, 2024-09-17T00:24:48.664 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-17T00:24:48.664 INFO:journalctl@ceph.mon.b.smithi099.stdout: exec(code, run_globals) 2024-09-17T00:24:48.664 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5579, in 2024-09-17T00:24:48.665 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5567, in main 2024-09-17T00:24:48.665 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3049, in command_deploy_from 2024-09-17T00:24:48.665 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3084, in _common_deploy 2024-09-17T00:24:48.665 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3104, in _deploy_daemon_container 2024-09-17T00:24:48.665 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 1124, in deploy_daemon 2024-09-17T00:24:48.665 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-17T00:24:48.666 INFO:journalctl@ceph.mon.b.smithi099.stdout: RuntimeError: Failed command: systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem: Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:24:48.666 INFO:journalctl@ceph.mon.b.smithi099.stdout: See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:24:48.666 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:48 smithi099 ceph-mon[56712]: Reconfiguring mon.b (monmap changed)... 2024-09-17T00:24:48.666 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:48 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-17T00:24:48.666 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:48 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-17T00:24:48.666 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:48 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:24:48.666 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:48 smithi099 ceph-mon[56712]: Reconfiguring daemon mon.b on smithi099 2024-09-17T00:24:48.853 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-09-17T00:24:48.853 INFO:teuthology.orchestra.run.smithi002.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f", 2024-09-17T00:24:48.853 INFO:teuthology.orchestra.run.smithi002.stdout: "in_progress": true, 2024-09-17T00:24:48.853 INFO:teuthology.orchestra.run.smithi002.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-17T00:24:48.854 INFO:teuthology.orchestra.run.smithi002.stdout: "services_complete": [ 2024-09-17T00:24:48.854 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr", 2024-09-17T00:24:48.854 INFO:teuthology.orchestra.run.smithi002.stdout: "mon" 2024-09-17T00:24:48.854 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-09-17T00:24:48.854 INFO:teuthology.orchestra.run.smithi002.stdout: "progress": "5/23 daemons upgraded", 2024-09-17T00:24:48.854 INFO:teuthology.orchestra.run.smithi002.stdout: "message": "Currently upgrading mon daemons", 2024-09-17T00:24:48.854 INFO:teuthology.orchestra.run.smithi002.stdout: "is_paused": false 2024-09-17T00:24:48.854 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-09-17T00:24:48.929 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:24:48 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[52757]: ts=2024-09-17T00:24:48.660Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd name=CephOSDFlapping index=13 msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.2\", device_class=\"ssd\", hostname=\"smithi002\", instance=\"172.21.15.99:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.2\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"3aad27a2-7489-11ef-bceb-c7b262605968\", cluster_addr=\"172.21.15.2\", device_class=\"ssd\", hostname=\"smithi002\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.2\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-17T00:24:49.284 INFO:teuthology.orchestra.run.smithi002.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2024-09-17T00:24:49.284 INFO:teuthology.orchestra.run.smithi002.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-09-17T00:24:49.284 INFO:teuthology.orchestra.run.smithi002.stdout: daemon iscsi.foo.smithi002.vmouem on smithi002 is in unknown state 2024-09-17T00:24:49.498 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:49 smithi099 ceph-mon[56712]: from='client.34320 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:24:49.498 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:49 smithi099 ceph-mon[56712]: from='client.34326 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:24:49.498 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:49 smithi099 ceph-mon[56712]: from='client.? 172.21.15.2:0/855392356' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:24:49.499 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:49 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:49.499 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:49 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:49.499 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:49 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-17T00:24:49.499 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:49 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-17T00:24:49.499 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:49 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-17T00:24:49.499 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:49 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:24:49.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:49 smithi002 ceph-mon[63804]: from='client.34320 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:24:49.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:49 smithi002 ceph-mon[63804]: from='client.34326 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:24:49.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:49 smithi002 ceph-mon[63804]: from='client.? 172.21.15.2:0/855392356' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:24:49.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:49 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:49.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:49 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:49.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:49 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-17T00:24:49.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:49 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-17T00:24:49.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:49 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-17T00:24:49.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:49 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:24:49.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:49 smithi002 ceph-mon[66732]: from='client.34320 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:24:49.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:49 smithi002 ceph-mon[66732]: from='client.34326 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:24:49.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:49 smithi002 ceph-mon[66732]: from='client.? 172.21.15.2:0/855392356' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:24:49.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:49 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:49.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:49 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:49.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:49 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-17T00:24:49.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:49 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-17T00:24:49.731 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:49 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-17T00:24:49.731 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:49 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:24:50.378 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:50 smithi099 ceph-mon[56712]: from='client.44245 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:24:50.378 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:50 smithi099 ceph-mon[56712]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:24:50.378 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:50 smithi099 ceph-mon[56712]: Reconfiguring mgr.x (monmap changed)... 2024-09-17T00:24:50.378 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:50 smithi099 ceph-mon[56712]: Reconfiguring daemon mgr.x on smithi099 2024-09-17T00:24:50.379 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:50 smithi099 ceph-mon[56712]: from='client.? 172.21.15.2:0/306655622' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T00:24:50.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:50 smithi002 ceph-mon[66732]: from='client.44245 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:24:50.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:50 smithi002 ceph-mon[66732]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:24:50.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:50 smithi002 ceph-mon[66732]: Reconfiguring mgr.x (monmap changed)... 2024-09-17T00:24:50.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:50 smithi002 ceph-mon[66732]: Reconfiguring daemon mgr.x on smithi099 2024-09-17T00:24:50.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:50 smithi002 ceph-mon[66732]: from='client.? 172.21.15.2:0/306655622' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T00:24:50.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:50 smithi002 ceph-mon[63804]: from='client.44245 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:24:50.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:50 smithi002 ceph-mon[63804]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:24:50.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:50 smithi002 ceph-mon[63804]: Reconfiguring mgr.x (monmap changed)... 2024-09-17T00:24:50.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:50 smithi002 ceph-mon[63804]: Reconfiguring daemon mgr.x on smithi099 2024-09-17T00:24:50.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:50 smithi002 ceph-mon[63804]: from='client.? 172.21.15.2:0/306655622' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T00:24:51.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:51 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:51.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:51 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:51.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:51 smithi099 ceph-mon[56712]: Reconfiguring osd.4 (monmap changed)... 2024-09-17T00:24:51.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:51 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-09-17T00:24:51.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:51 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:24:51.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:51 smithi099 ceph-mon[56712]: Reconfiguring daemon osd.4 on smithi099 2024-09-17T00:24:51.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:51 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:51.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:51 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:51.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:51 smithi002 ceph-mon[66732]: Reconfiguring osd.4 (monmap changed)... 2024-09-17T00:24:51.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:51 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-09-17T00:24:51.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:51 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:24:51.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:51 smithi002 ceph-mon[66732]: Reconfiguring daemon osd.4 on smithi099 2024-09-17T00:24:51.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:51 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:51.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:51 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:51.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:51 smithi002 ceph-mon[63804]: Reconfiguring osd.4 (monmap changed)... 2024-09-17T00:24:51.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:51 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-09-17T00:24:51.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:51 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:24:51.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:51 smithi002 ceph-mon[63804]: Reconfiguring daemon osd.4 on smithi099 2024-09-17T00:24:52.550 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:52 smithi099 ceph-mon[56712]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-17T00:24:52.550 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:52 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:52.550 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:52 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:52.550 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:52 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-09-17T00:24:52.551 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:52 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:24:52.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:52 smithi002 ceph-mon[66732]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-17T00:24:52.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:52 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:52.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:52 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:52.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:52 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-09-17T00:24:52.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:52 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:24:52.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:52 smithi002 ceph-mon[63804]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-17T00:24:52.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:52 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:52.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:52 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:52.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:52 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-09-17T00:24:52.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:52 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:24:53.717 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:24:53 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[52757]: ts=2024-09-17T00:24:53.505Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes name=CephNodeDiskspaceWarning index=4 msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi002\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi002\", job=\"node\", machine=\"x86_64\", nodename=\"smithi002\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"3aad27a2-7489-11ef-bceb-c7b262605968\", domainname=\"(none)\", instance=\"smithi002\", job=\"node\", machine=\"x86_64\", nodename=\"smithi002\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-17T00:24:53.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:53 smithi099 ceph-mon[56712]: Reconfiguring osd.5 (monmap changed)... 2024-09-17T00:24:53.718 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:53 smithi099 ceph-mon[56712]: Reconfiguring daemon osd.5 on smithi099 2024-09-17T00:24:53.718 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:53 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:53.718 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:53 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:53.718 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:53 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-09-17T00:24:53.718 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:53 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:24:53.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:53 smithi002 ceph-mon[63804]: Reconfiguring osd.5 (monmap changed)... 2024-09-17T00:24:53.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:53 smithi002 ceph-mon[63804]: Reconfiguring daemon osd.5 on smithi099 2024-09-17T00:24:53.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:53 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:53.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:53 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:53.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:53 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-09-17T00:24:53.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:53 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:24:53.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:53 smithi002 ceph-mon[66732]: Reconfiguring osd.5 (monmap changed)... 2024-09-17T00:24:53.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:53 smithi002 ceph-mon[66732]: Reconfiguring daemon osd.5 on smithi099 2024-09-17T00:24:53.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:53 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:53.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:53 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:53.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:53 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-09-17T00:24:53.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:53 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:24:54.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:54 smithi099 ceph-mon[56712]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-17T00:24:54.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:54 smithi099 ceph-mon[56712]: Reconfiguring osd.6 (monmap changed)... 2024-09-17T00:24:54.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:54 smithi099 ceph-mon[56712]: Reconfiguring daemon osd.6 on smithi099 2024-09-17T00:24:54.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:54 smithi002 ceph-mon[63804]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-17T00:24:54.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:54 smithi002 ceph-mon[63804]: Reconfiguring osd.6 (monmap changed)... 2024-09-17T00:24:54.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:54 smithi002 ceph-mon[63804]: Reconfiguring daemon osd.6 on smithi099 2024-09-17T00:24:54.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:54 smithi002 ceph-mon[66732]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-17T00:24:54.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:54 smithi002 ceph-mon[66732]: Reconfiguring osd.6 (monmap changed)... 2024-09-17T00:24:54.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:54 smithi002 ceph-mon[66732]: Reconfiguring daemon osd.6 on smithi099 2024-09-17T00:24:56.141 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:55 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:56.141 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:55 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:56.141 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:55 smithi099 ceph-mon[56712]: Reconfiguring osd.7 (monmap changed)... 2024-09-17T00:24:56.142 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:55 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-09-17T00:24:56.142 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:55 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:24:56.142 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:55 smithi099 ceph-mon[56712]: Reconfiguring daemon osd.7 on smithi099 2024-09-17T00:24:56.142 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:55 smithi099 ceph-mon[56712]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-09-17T00:24:56.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:55 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:56.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:55 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:56.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:55 smithi002 ceph-mon[63804]: Reconfiguring osd.7 (monmap changed)... 2024-09-17T00:24:56.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:55 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-09-17T00:24:56.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:55 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:24:56.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:55 smithi002 ceph-mon[63804]: Reconfiguring daemon osd.7 on smithi099 2024-09-17T00:24:56.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:55 smithi002 ceph-mon[63804]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-09-17T00:24:56.230 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:55 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:56.230 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:55 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:56.230 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:55 smithi002 ceph-mon[66732]: Reconfiguring osd.7 (monmap changed)... 2024-09-17T00:24:56.230 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:55 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-09-17T00:24:56.230 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:55 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:24:56.230 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:55 smithi002 ceph-mon[66732]: Reconfiguring daemon osd.7 on smithi099 2024-09-17T00:24:56.230 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:55 smithi002 ceph-mon[66732]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-09-17T00:24:56.979 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:24:56 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: ::ffff:172.21.15.99 - - [17/Sep/2024:00:24:56] "GET /metrics HTTP/1.1" 200 38257 "" "Prometheus/2.43.0" 2024-09-17T00:24:57.432 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:57 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:57.432 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:57 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:57.432 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:57 smithi099 ceph-mon[56712]: Reconfiguring rgw.foo.smithi099.utzfap (monmap changed)... 2024-09-17T00:24:57.432 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:57 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi099.utzfap", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T00:24:57.432 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:57 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi099.utzfap", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T00:24:57.432 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:57 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:24:57.432 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:57 smithi099 ceph-mon[56712]: Reconfiguring daemon rgw.foo.smithi099.utzfap on smithi099 2024-09-17T00:24:57.432 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:57 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T00:24:57.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:57 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:57.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:57 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:57.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:57 smithi002 ceph-mon[63804]: Reconfiguring rgw.foo.smithi099.utzfap (monmap changed)... 2024-09-17T00:24:57.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:57 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi099.utzfap", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T00:24:57.480 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:57 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi099.utzfap", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T00:24:57.480 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:57 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:24:57.480 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:57 smithi002 ceph-mon[63804]: Reconfiguring daemon rgw.foo.smithi099.utzfap on smithi099 2024-09-17T00:24:57.480 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:57 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T00:24:57.480 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:57 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:57.480 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:57 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:57.480 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:57 smithi002 ceph-mon[66732]: Reconfiguring rgw.foo.smithi099.utzfap (monmap changed)... 2024-09-17T00:24:57.480 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:57 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi099.utzfap", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T00:24:57.480 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:57 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi099.utzfap", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T00:24:57.480 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:57 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:24:57.480 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:57 smithi002 ceph-mon[66732]: Reconfiguring daemon rgw.foo.smithi099.utzfap on smithi099 2024-09-17T00:24:57.481 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:57 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T00:24:58.382 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:58 smithi002 ceph-mon[63804]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-09-17T00:24:58.382 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:58 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:58.382 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:58 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:58.382 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:58 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi099.nzvkpm", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T00:24:58.382 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:58 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi099.nzvkpm", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T00:24:58.382 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:58 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:24:58.382 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:58 smithi002 ceph-mon[66732]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-09-17T00:24:58.382 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:58 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:58.382 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:58 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:58.383 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:58 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi099.nzvkpm", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T00:24:58.383 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:58 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi099.nzvkpm", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T00:24:58.383 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:58 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:24:58.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:58 smithi099 ceph-mon[56712]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-09-17T00:24:58.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:58 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:58.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:58 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:58.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:58 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi099.nzvkpm", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T00:24:58.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:58 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi099.nzvkpm", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T00:24:58.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:58 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:24:58.855 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:24:58 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[52757]: ts=2024-09-17T00:24:58.659Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd name=CephOSDFlapping index=13 msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.2\", device_class=\"ssd\", hostname=\"smithi002\", instance=\"172.21.15.99:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.2\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"3aad27a2-7489-11ef-bceb-c7b262605968\", cluster_addr=\"172.21.15.2\", device_class=\"ssd\", hostname=\"smithi002\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.2\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-17T00:24:59.463 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:59 smithi002 ceph-mon[66732]: Reconfiguring rgw.smpl.smithi099.nzvkpm (monmap changed)... 2024-09-17T00:24:59.464 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:59 smithi002 ceph-mon[66732]: Reconfiguring daemon rgw.smpl.smithi099.nzvkpm on smithi099 2024-09-17T00:24:59.464 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:59 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:59.464 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:59 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:59.464 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:59 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T00:24:59.464 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:59 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:24:59.464 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:24:59 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:24:59.464 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:59 smithi002 ceph-mon[63804]: Reconfiguring rgw.smpl.smithi099.nzvkpm (monmap changed)... 2024-09-17T00:24:59.464 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:59 smithi002 ceph-mon[63804]: Reconfiguring daemon rgw.smpl.smithi099.nzvkpm on smithi099 2024-09-17T00:24:59.465 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:59 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:59.465 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:59 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:59.465 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:59 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T00:24:59.465 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:59 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:24:59.465 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:24:59 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:24:59.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:59 smithi099 ceph-mon[56712]: Reconfiguring rgw.smpl.smithi099.nzvkpm (monmap changed)... 2024-09-17T00:24:59.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:59 smithi099 ceph-mon[56712]: Reconfiguring daemon rgw.smpl.smithi099.nzvkpm on smithi099 2024-09-17T00:24:59.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:59 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:59.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:59 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:24:59.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:59 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T00:24:59.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:59 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:24:59.468 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:24:59 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:25:00.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:25:00 smithi099 ceph-mon[56712]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-09-17T00:25:00.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:25:00 smithi099 ceph-mon[56712]: Upgrade: Setting container_image for all mon 2024-09-17T00:25:00.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:25:00 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:25:00.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:25:00 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-09-17T00:25:00.718 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:25:00 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-09-17T00:25:00.718 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:25:00 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]': finished 2024-09-17T00:25:00.718 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:25:00 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-09-17T00:25:00.718 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:25:00 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-09-17T00:25:00.718 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:25:00 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]': finished 2024-09-17T00:25:00.718 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:25:00 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-09-17T00:25:00.718 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:25:00 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-09-17T00:25:00.718 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:25:00 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]': finished 2024-09-17T00:25:00.718 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:25:00 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:25:00.718 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:25:00 smithi099 ceph-mon[56712]: Upgrade: Setting container_image for all crash 2024-09-17T00:25:00.718 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:25:00 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:25:00.718 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:25:00 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-09-17T00:25:00.719 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:25:00 smithi099 ceph-mon[56712]: from='mon.1 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-09-17T00:25:00.719 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:25:00 smithi099 ceph-mon[56712]: Upgrade: osd.0 is safe to restart 2024-09-17T00:25:00.719 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:25:00 smithi099 ceph-mon[56712]: from='client.? 172.21.15.2:0/1582274113' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-17T00:25:00.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:25:00 smithi002 ceph-mon[63804]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-09-17T00:25:00.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:25:00 smithi002 ceph-mon[63804]: Upgrade: Setting container_image for all mon 2024-09-17T00:25:00.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:25:00 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:25:00.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:25:00 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-09-17T00:25:00.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:25:00 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-09-17T00:25:00.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:25:00 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]': finished 2024-09-17T00:25:00.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:25:00 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-09-17T00:25:00.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:25:00 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-09-17T00:25:00.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:25:00 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]': finished 2024-09-17T00:25:00.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:25:00 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-09-17T00:25:00.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:25:00 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-09-17T00:25:00.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:25:00 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]': finished 2024-09-17T00:25:00.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:25:00 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:25:00.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:25:00 smithi002 ceph-mon[63804]: Upgrade: Setting container_image for all crash 2024-09-17T00:25:00.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:25:00 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:25:00.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:25:00 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-09-17T00:25:00.731 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:25:00 smithi002 ceph-mon[63804]: from='mon.1 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-09-17T00:25:00.731 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:25:00 smithi002 ceph-mon[63804]: Upgrade: osd.0 is safe to restart 2024-09-17T00:25:00.731 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:25:00 smithi002 ceph-mon[63804]: from='client.? 172.21.15.2:0/1582274113' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-17T00:25:00.731 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:25:00 smithi002 ceph-mon[66732]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-09-17T00:25:00.731 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:25:00 smithi002 ceph-mon[66732]: Upgrade: Setting container_image for all mon 2024-09-17T00:25:00.731 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:25:00 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:25:00.731 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:25:00 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-09-17T00:25:00.731 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:25:00 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-09-17T00:25:00.731 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:25:00 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]': finished 2024-09-17T00:25:00.731 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:25:00 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-09-17T00:25:00.731 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:25:00 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-09-17T00:25:00.732 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:25:00 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]': finished 2024-09-17T00:25:00.732 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:25:00 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-09-17T00:25:00.732 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:25:00 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-09-17T00:25:00.732 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:25:00 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]': finished 2024-09-17T00:25:00.732 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:25:00 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:25:00.732 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:25:00 smithi002 ceph-mon[66732]: Upgrade: Setting container_image for all crash 2024-09-17T00:25:00.732 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:25:00 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:25:00.732 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:25:00 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-09-17T00:25:00.732 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:25:00 smithi002 ceph-mon[66732]: from='mon.1 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-09-17T00:25:00.732 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:25:00 smithi002 ceph-mon[66732]: Upgrade: osd.0 is safe to restart 2024-09-17T00:25:00.732 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:25:00 smithi002 ceph-mon[66732]: from='client.? 172.21.15.2:0/1582274113' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-17T00:25:02.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:25:01 smithi099 ceph-mon[56712]: Upgrade: Updating osd.0 2024-09-17T00:25:02.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:25:01 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:25:02.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:25:01 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-09-17T00:25:02.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:25:01 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:25:02.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:25:01 smithi099 ceph-mon[56712]: Deploying daemon osd.0 on smithi002 2024-09-17T00:25:02.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:25:01 smithi099 ceph-mon[56712]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:25:02.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:25:01 smithi002 ceph-mon[66732]: Upgrade: Updating osd.0 2024-09-17T00:25:02.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:25:01 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:25:02.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:25:01 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-09-17T00:25:02.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:25:01 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:25:02.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:25:01 smithi002 ceph-mon[66732]: Deploying daemon osd.0 on smithi002 2024-09-17T00:25:02.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:25:01 smithi002 ceph-mon[66732]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:25:02.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:25:01 smithi002 ceph-mon[63804]: Upgrade: Updating osd.0 2024-09-17T00:25:02.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:25:01 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:25:02.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:25:01 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-09-17T00:25:02.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:25:01 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:25:02.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:25:01 smithi002 ceph-mon[63804]: Deploying daemon osd.0 on smithi002 2024-09-17T00:25:02.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:25:01 smithi002 ceph-mon[63804]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:25:03.921 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:25:03 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[52757]: ts=2024-09-17T00:25:03.506Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes name=CephNodeDiskspaceWarning index=4 msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi002\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi002\", job=\"node\", machine=\"x86_64\", nodename=\"smithi002\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"3aad27a2-7489-11ef-bceb-c7b262605968\", domainname=\"(none)\", instance=\"smithi002\", job=\"node\", machine=\"x86_64\", nodename=\"smithi002\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-17T00:25:03.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:25:03 smithi002 ceph-mon[66732]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-09-17T00:25:03.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:25:03 smithi002 ceph-mon[63804]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-09-17T00:25:04.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:25:03 smithi099 ceph-mon[56712]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-09-17T00:25:04.479 INFO:journalctl@ceph.osd.0.smithi002.stdout:Sep 17 00:25:04 smithi002 systemd[1]: Stopping Ceph osd.0 for 3aad27a2-7489-11ef-bceb-c7b262605968... 2024-09-17T00:25:04.479 INFO:journalctl@ceph.osd.0.smithi002.stdout:Sep 17 00:25:04 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-0[31022]: 2024-09-17T00:25:04.306+0000 7f6fcfef9700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-17T00:25:04.479 INFO:journalctl@ceph.osd.0.smithi002.stdout:Sep 17 00:25:04 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-0[31022]: 2024-09-17T00:25:04.306+0000 7f6fcfef9700 -1 osd.0 112 *** Got signal Terminated *** 2024-09-17T00:25:04.479 INFO:journalctl@ceph.osd.0.smithi002.stdout:Sep 17 00:25:04 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-0[31022]: 2024-09-17T00:25:04.306+0000 7f6fcfef9700 -1 osd.0 112 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-17T00:25:05.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:25:04 smithi099 ceph-mon[56712]: osd.0 marked itself down and dead 2024-09-17T00:25:05.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:25:04 smithi002 ceph-mon[63804]: osd.0 marked itself down and dead 2024-09-17T00:25:05.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:25:04 smithi002 ceph-mon[66732]: osd.0 marked itself down and dead 2024-09-17T00:25:05.597 INFO:journalctl@ceph.osd.0.smithi002.stdout:Sep 17 00:25:05 smithi002 podman[70789]: 2024-09-17 00:25:05.320511161 +0000 UTC m=+1.113690740 container died a0ad0fb7aa345ec0afcb140d337cc04251776723b5ee79c156c962d11de38975 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-0, version=8, GIT_CLEAN=True, com.redhat.component=centos-stream-container, build-date=2022-05-03T08:36:31.336870, ceph=True, io.buildah.version=1.19.8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, RELEASE=HEAD, vendor=Red Hat, Inc., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux , release=754, io.openshift.tags=base centos centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.display-name=CentOS Stream 8, architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=centos-stream, distribution-scope=public, CEPH_POINT_RELEASE=-17.2.0, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_BRANCH=HEAD, io.openshift.expose-services=, vcs-type=git) 2024-09-17T00:25:05.919 INFO:journalctl@ceph.osd.0.smithi002.stdout:Sep 17 00:25:05 smithi002 podman[70789]: 2024-09-17 00:25:05.707080991 +0000 UTC m=+1.500260560 container cleanup a0ad0fb7aa345ec0afcb140d337cc04251776723b5ee79c156c962d11de38975 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-0, name=centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.component=centos-stream-container, io.k8s.display-name=CentOS Stream 8, CEPH_POINT_RELEASE=-17.2.0, io.buildah.version=1.19.8, GIT_BRANCH=HEAD, version=8, vcs-type=git, vendor=Red Hat, Inc., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, RELEASE=HEAD, build-date=2022-05-03T08:36:31.336870, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_CLEAN=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.tags=base centos centos-stream, io.openshift.expose-services=, release=754, distribution-scope=public, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, maintainer=Guillaume Abrioux , architecture=x86_64) 2024-09-17T00:25:05.920 INFO:journalctl@ceph.osd.0.smithi002.stdout:Sep 17 00:25:05 smithi002 bash[70789]: ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-0 2024-09-17T00:25:06.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:25:05 smithi099 ceph-mon[56712]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-09-17T00:25:06.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:25:05 smithi099 ceph-mon[56712]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-17T00:25:06.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:25:05 smithi099 ceph-mon[56712]: osdmap e113: 8 total, 7 up, 8 in 2024-09-17T00:25:06.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:25:05 smithi002 ceph-mon[63804]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-09-17T00:25:06.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:25:05 smithi002 ceph-mon[63804]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-17T00:25:06.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:25:05 smithi002 ceph-mon[63804]: osdmap e113: 8 total, 7 up, 8 in 2024-09-17T00:25:06.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:25:05 smithi002 ceph-mon[66732]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-09-17T00:25:06.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:25:05 smithi002 ceph-mon[66732]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-17T00:25:06.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:25:05 smithi002 ceph-mon[66732]: osdmap e113: 8 total, 7 up, 8 in 2024-09-17T00:25:06.229 INFO:journalctl@ceph.osd.0.smithi002.stdout:Sep 17 00:25:05 smithi002 podman[70803]: 2024-09-17 00:25:05.983297154 +0000 UTC m=+0.660068430 container remove a0ad0fb7aa345ec0afcb140d337cc04251776723b5ee79c156c962d11de38975 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-0, com.redhat.component=centos-stream-container, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, ceph=True, name=centos-stream, vendor=Red Hat, Inc., RELEASE=HEAD, vcs-type=git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, release=754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_BRANCH=HEAD, distribution-scope=public, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, version=8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.buildah.version=1.19.8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.display-name=CentOS Stream 8, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.openshift.tags=base centos centos-stream, build-date=2022-05-03T08:36:31.336870, CEPH_POINT_RELEASE=-17.2.0, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.expose-services=) 2024-09-17T00:25:06.480 INFO:journalctl@ceph.osd.0.smithi002.stdout:Sep 17 00:25:06 smithi002 podman[70893]: 2024-09-17 00:25:06.239224656 +0000 UTC m=+0.017230282 image pull ebb0615149def4a3179c82c0cc47af75808300abb9f1142788e5e605a7ac0e5b quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f 2024-09-17T00:25:06.481 INFO:journalctl@ceph.osd.0.smithi002.stdout:Sep 17 00:25:06 smithi002 podman[70893]: 2024-09-17 00:25:06.367633659 +0000 UTC m=+0.145639274 container create ff25186b254046ee849f7dc05cfb8c3b85e0a8e931034554f0bbb3d1b6f8d4d2 (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-0-deactivate, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, RELEASE=squid-c2ddf97, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-17T00:25:06.787 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:25:06 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: ::ffff:172.21.15.99 - - [17/Sep/2024:00:25:06] "GET /metrics HTTP/1.1" 200 38260 "" "Prometheus/2.43.0" 2024-09-17T00:25:06.787 INFO:journalctl@ceph.osd.0.smithi002.stdout:Sep 17 00:25:06 smithi002 podman[70893]: 2024-09-17 00:25:06.587040852 +0000 UTC m=+0.365046468 container init ff25186b254046ee849f7dc05cfb8c3b85e0a8e931034554f0bbb3d1b6f8d4d2 (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-0-deactivate, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, RELEASE=squid-c2ddf97) 2024-09-17T00:25:06.787 INFO:journalctl@ceph.osd.0.smithi002.stdout:Sep 17 00:25:06 smithi002 podman[70893]: 2024-09-17 00:25:06.590752399 +0000 UTC m=+0.368758015 container start ff25186b254046ee849f7dc05cfb8c3b85e0a8e931034554f0bbb3d1b6f8d4d2 (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-0-deactivate, RELEASE=squid-c2ddf97, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-17T00:25:06.787 INFO:journalctl@ceph.osd.0.smithi002.stdout:Sep 17 00:25:06 smithi002 podman[70893]: 2024-09-17 00:25:06.661616245 +0000 UTC m=+0.439621881 container attach ff25186b254046ee849f7dc05cfb8c3b85e0a8e931034554f0bbb3d1b6f8d4d2 (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-0-deactivate, RELEASE=squid-c2ddf97, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240731, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-09-17T00:25:07.078 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:25:06 smithi002 ceph-mon[66732]: osdmap e114: 8 total, 7 up, 8 in 2024-09-17T00:25:07.078 INFO:journalctl@ceph.osd.0.smithi002.stdout:Sep 17 00:25:06 smithi002 podman[70893]: 2024-09-17 00:25:06.79063782 +0000 UTC m=+0.568643444 container died ff25186b254046ee849f7dc05cfb8c3b85e0a8e931034554f0bbb3d1b6f8d4d2 (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-0-deactivate, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=squid-c2ddf97, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731) 2024-09-17T00:25:07.078 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:25:06 smithi002 ceph-mon[63804]: osdmap e114: 8 total, 7 up, 8 in 2024-09-17T00:25:07.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:25:06 smithi099 ceph-mon[56712]: osdmap e114: 8 total, 7 up, 8 in 2024-09-17T00:25:07.729 INFO:journalctl@ceph.osd.0.smithi002.stdout:Sep 17 00:25:07 smithi002 podman[70916]: 2024-09-17 00:25:07.373945078 +0000 UTC m=+0.573267147 container remove ff25186b254046ee849f7dc05cfb8c3b85e0a8e931034554f0bbb3d1b6f8d4d2 (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-0-deactivate, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=squid-c2ddf97) 2024-09-17T00:25:07.729 INFO:journalctl@ceph.osd.0.smithi002.stdout:Sep 17 00:25:07 smithi002 systemd[1]: ceph-3aad27a2-7489-11ef-bceb-c7b262605968@osd.0.service: Deactivated successfully. 2024-09-17T00:25:07.729 INFO:journalctl@ceph.osd.0.smithi002.stdout:Sep 17 00:25:07 smithi002 systemd[1]: Stopped Ceph osd.0 for 3aad27a2-7489-11ef-bceb-c7b262605968. 2024-09-17T00:25:07.730 INFO:journalctl@ceph.osd.0.smithi002.stdout:Sep 17 00:25:07 smithi002 systemd[1]: ceph-3aad27a2-7489-11ef-bceb-c7b262605968@osd.0.service: Consumed 8.178s CPU time. 2024-09-17T00:25:07.730 INFO:journalctl@ceph.osd.0.smithi002.stdout:Sep 17 00:25:07 smithi002 systemd[1]: Starting Ceph osd.0 for 3aad27a2-7489-11ef-bceb-c7b262605968... 2024-09-17T00:25:08.124 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:25:07 smithi002 ceph-mon[66732]: pgmap v56: 161 pgs: 21 stale+active+clean, 140 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-09-17T00:25:08.125 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:25:07 smithi002 ceph-mon[63804]: pgmap v56: 161 pgs: 21 stale+active+clean, 140 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-09-17T00:25:08.125 INFO:journalctl@ceph.osd.0.smithi002.stdout:Sep 17 00:25:07 smithi002 podman[71016]: 2024-09-17 00:25:07.716581554 +0000 UTC m=+0.017397318 image pull ebb0615149def4a3179c82c0cc47af75808300abb9f1142788e5e605a7ac0e5b quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f 2024-09-17T00:25:08.125 INFO:journalctl@ceph.osd.0.smithi002.stdout:Sep 17 00:25:07 smithi002 podman[71016]: 2024-09-17 00:25:07.858833016 +0000 UTC m=+0.159648777 container create 79459b33440c39851422c78a294c09f348033038b3a56e6bdb7625781736f231 (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-0-activate, RELEASE=squid-c2ddf97, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-09-17T00:25:08.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:25:07 smithi099 ceph-mon[56712]: pgmap v56: 161 pgs: 21 stale+active+clean, 140 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-09-17T00:25:08.479 INFO:journalctl@ceph.osd.0.smithi002.stdout:Sep 17 00:25:08 smithi002 podman[71016]: 2024-09-17 00:25:08.127982879 +0000 UTC m=+0.428798656 container init 79459b33440c39851422c78a294c09f348033038b3a56e6bdb7625781736f231 (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-0-activate, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, RELEASE=squid-c2ddf97, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux ) 2024-09-17T00:25:08.479 INFO:journalctl@ceph.osd.0.smithi002.stdout:Sep 17 00:25:08 smithi002 podman[71016]: 2024-09-17 00:25:08.131812737 +0000 UTC m=+0.432628499 container start 79459b33440c39851422c78a294c09f348033038b3a56e6bdb7625781736f231 (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-0-activate, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=squid-c2ddf97, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731) 2024-09-17T00:25:08.479 INFO:journalctl@ceph.osd.0.smithi002.stdout:Sep 17 00:25:08 smithi002 podman[71016]: 2024-09-17 00:25:08.228043195 +0000 UTC m=+0.528858968 container attach 79459b33440c39851422c78a294c09f348033038b3a56e6bdb7625781736f231 (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-0-activate, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, io.buildah.version=1.37.2, RELEASE=squid-c2ddf97, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-17T00:25:08.480 INFO:journalctl@ceph.osd.0.smithi002.stdout:Sep 17 00:25:08 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-0-activate[71031]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-17T00:25:08.480 INFO:journalctl@ceph.osd.0.smithi002.stdout:Sep 17 00:25:08 smithi002 bash[71016]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-17T00:25:08.480 INFO:journalctl@ceph.osd.0.smithi002.stdout:Sep 17 00:25:08 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-0-activate[71031]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-17T00:25:08.480 INFO:journalctl@ceph.osd.0.smithi002.stdout:Sep 17 00:25:08 smithi002 bash[71016]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-17T00:25:08.967 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:25:08 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[52757]: ts=2024-09-17T00:25:08.660Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd name=CephOSDFlapping index=13 msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.2\", device_class=\"ssd\", hostname=\"smithi002\", instance=\"172.21.15.99:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.2\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"3aad27a2-7489-11ef-bceb-c7b262605968\", cluster_addr=\"172.21.15.2\", device_class=\"ssd\", hostname=\"smithi002\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.2\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-17T00:25:09.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:25:08 smithi099 ceph-mon[56712]: Health check failed: Degraded data redundancy: 74/726 objects degraded (10.193%), 20 pgs degraded (PG_DEGRADED) 2024-09-17T00:25:09.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:25:08 smithi002 ceph-mon[63804]: Health check failed: Degraded data redundancy: 74/726 objects degraded (10.193%), 20 pgs degraded (PG_DEGRADED) 2024-09-17T00:25:09.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:25:08 smithi002 ceph-mon[66732]: Health check failed: Degraded data redundancy: 74/726 objects degraded (10.193%), 20 pgs degraded (PG_DEGRADED) 2024-09-17T00:25:09.959 INFO:journalctl@ceph.osd.0.smithi002.stdout:Sep 17 00:25:09 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-0-activate[71031]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-17T00:25:09.960 INFO:journalctl@ceph.osd.0.smithi002.stdout:Sep 17 00:25:09 smithi002 bash[71016]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-17T00:25:09.960 INFO:journalctl@ceph.osd.0.smithi002.stdout:Sep 17 00:25:09 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-0-activate[71031]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-09-17T00:25:09.960 INFO:journalctl@ceph.osd.0.smithi002.stdout:Sep 17 00:25:09 smithi002 bash[71016]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-09-17T00:25:09.960 INFO:journalctl@ceph.osd.0.smithi002.stdout:Sep 17 00:25:09 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-0-activate[71031]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-09-17T00:25:09.960 INFO:journalctl@ceph.osd.0.smithi002.stdout:Sep 17 00:25:09 smithi002 bash[71016]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-09-17T00:25:10.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:25:09 smithi002 ceph-mon[66732]: pgmap v57: 161 pgs: 38 active+undersized, 20 active+undersized+degraded, 103 active+clean; 585 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s; 74/726 objects degraded (10.193%) 2024-09-17T00:25:10.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:25:09 smithi002 ceph-mon[66732]: from='client.44266 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:25:10.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:25:09 smithi002 ceph-mon[63804]: pgmap v57: 161 pgs: 38 active+undersized, 20 active+undersized+degraded, 103 active+clean; 585 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s; 74/726 objects degraded (10.193%) 2024-09-17T00:25:10.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:25:09 smithi002 ceph-mon[63804]: from='client.44266 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:25:10.230 INFO:journalctl@ceph.osd.0.smithi002.stdout:Sep 17 00:25:09 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-0-activate[71031]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-09-17T00:25:10.230 INFO:journalctl@ceph.osd.0.smithi002.stdout:Sep 17 00:25:09 smithi002 bash[71016]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-09-17T00:25:10.230 INFO:journalctl@ceph.osd.0.smithi002.stdout:Sep 17 00:25:09 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-0-activate[71031]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-09-17T00:25:10.230 INFO:journalctl@ceph.osd.0.smithi002.stdout:Sep 17 00:25:09 smithi002 bash[71016]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-09-17T00:25:10.230 INFO:journalctl@ceph.osd.0.smithi002.stdout:Sep 17 00:25:09 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-0-activate[71031]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-17T00:25:10.230 INFO:journalctl@ceph.osd.0.smithi002.stdout:Sep 17 00:25:09 smithi002 bash[71016]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-17T00:25:10.230 INFO:journalctl@ceph.osd.0.smithi002.stdout:Sep 17 00:25:09 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-0-activate[71031]: --> ceph-volume raw activate successful for osd ID: 0 2024-09-17T00:25:10.230 INFO:journalctl@ceph.osd.0.smithi002.stdout:Sep 17 00:25:09 smithi002 bash[71016]: --> ceph-volume raw activate successful for osd ID: 0 2024-09-17T00:25:10.231 INFO:journalctl@ceph.osd.0.smithi002.stdout:Sep 17 00:25:09 smithi002 podman[71016]: 2024-09-17 00:25:09.999067346 +0000 UTC m=+2.299883114 container died 79459b33440c39851422c78a294c09f348033038b3a56e6bdb7625781736f231 (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-0-activate, RELEASE=squid-c2ddf97, org.label-schema.build-date=20240731, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-17T00:25:10.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:25:09 smithi099 ceph-mon[56712]: pgmap v57: 161 pgs: 38 active+undersized, 20 active+undersized+degraded, 103 active+clean; 585 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s; 74/726 objects degraded (10.193%) 2024-09-17T00:25:10.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:25:09 smithi099 ceph-mon[56712]: from='client.44266 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:25:10.729 INFO:journalctl@ceph.osd.0.smithi002.stdout:Sep 17 00:25:10 smithi002 podman[71165]: 2024-09-17 00:25:10.557045509 +0000 UTC m=+0.550940048 container remove 79459b33440c39851422c78a294c09f348033038b3a56e6bdb7625781736f231 (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-0-activate, org.label-schema.license=GPLv2, RELEASE=squid-c2ddf97, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-17T00:25:11.041 INFO:journalctl@ceph.osd.0.smithi002.stdout:Sep 17 00:25:10 smithi002 podman[71225]: 2024-09-17 00:25:10.679460879 +0000 UTC m=+0.017112211 image pull ebb0615149def4a3179c82c0cc47af75808300abb9f1142788e5e605a7ac0e5b quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f 2024-09-17T00:25:11.041 INFO:journalctl@ceph.osd.0.smithi002.stdout:Sep 17 00:25:10 smithi002 podman[71225]: 2024-09-17 00:25:10.791326995 +0000 UTC m=+0.128978315 container create 8345719cb6f0d21519d86b3f7fed22523f2ff0f9daa2ac0c3d61ccec8a7d4e8c (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, RELEASE=squid-c2ddf97, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2) 2024-09-17T00:25:11.041 INFO:journalctl@ceph.osd.0.smithi002.stdout:Sep 17 00:25:11 smithi002 podman[71225]: 2024-09-17 00:25:11.018547988 +0000 UTC m=+0.356199318 container init 8345719cb6f0d21519d86b3f7fed22523f2ff0f9daa2ac0c3d61ccec8a7d4e8c (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-0, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=squid-c2ddf97, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, ceph=True) 2024-09-17T00:25:11.041 INFO:journalctl@ceph.osd.0.smithi002.stdout:Sep 17 00:25:11 smithi002 podman[71225]: 2024-09-17 00:25:11.022068899 +0000 UTC m=+0.359720220 container start 8345719cb6f0d21519d86b3f7fed22523f2ff0f9daa2ac0c3d61ccec8a7d4e8c (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-0, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, ceph=True, RELEASE=squid-c2ddf97) 2024-09-17T00:25:11.041 INFO:journalctl@ceph.osd.0.smithi002.stdout:Sep 17 00:25:11 smithi002 ceph-osd[71244]: -- 172.21.15.2:0/61684583 <== mon.1 v2:172.21.15.2:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55c02db9da40 con 0x55c02e964400 2024-09-17T00:25:11.479 INFO:journalctl@ceph.osd.0.smithi002.stdout:Sep 17 00:25:11 smithi002 bash[71225]: 8345719cb6f0d21519d86b3f7fed22523f2ff0f9daa2ac0c3d61ccec8a7d4e8c 2024-09-17T00:25:11.480 INFO:journalctl@ceph.osd.0.smithi002.stdout:Sep 17 00:25:11 smithi002 systemd[1]: Started Ceph osd.0 for 3aad27a2-7489-11ef-bceb-c7b262605968. 2024-09-17T00:25:12.434 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:25:12 smithi002 ceph-mon[63804]: pgmap v58: 161 pgs: 38 active+undersized, 20 active+undersized+degraded, 103 active+clean; 585 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 74/726 objects degraded (10.193%) 2024-09-17T00:25:12.434 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:25:12 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:25:12.434 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:25:12 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:25:12.434 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:25:12 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:25:12.434 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:25:12 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T00:25:12.434 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:25:12 smithi002 ceph-mon[66732]: pgmap v58: 161 pgs: 38 active+undersized, 20 active+undersized+degraded, 103 active+clean; 585 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 74/726 objects degraded (10.193%) 2024-09-17T00:25:12.434 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:25:12 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:25:12.435 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:25:12 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:25:12.435 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:25:12 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:25:12.435 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:25:12 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T00:25:12.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:25:12 smithi099 ceph-mon[56712]: pgmap v58: 161 pgs: 38 active+undersized, 20 active+undersized+degraded, 103 active+clean; 585 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 74/726 objects degraded (10.193%) 2024-09-17T00:25:12.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:25:12 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:25:12.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:25:12 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:25:12.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:25:12 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:25:12.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:25:12 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T00:25:12.715 INFO:journalctl@ceph.osd.0.smithi002.stdout:Sep 17 00:25:12 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-0[71240]: 2024-09-17T00:25:12.436+0000 7f9894e32640 -1 Falling back to public interface 2024-09-17T00:25:13.967 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:25:13 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[52757]: ts=2024-09-17T00:25:13.505Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes name=CephNodeDiskspaceWarning index=4 msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi002\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi002\", job=\"node\", machine=\"x86_64\", nodename=\"smithi002\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"3aad27a2-7489-11ef-bceb-c7b262605968\", domainname=\"(none)\", instance=\"smithi002\", job=\"node\", machine=\"x86_64\", nodename=\"smithi002\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-17T00:25:14.320 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:25:14 smithi002 ceph-mon[63804]: pgmap v59: 161 pgs: 38 active+undersized, 20 active+undersized+degraded, 103 active+clean; 585 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 383 B/s rd, 0 op/s; 74/726 objects degraded (10.193%) 2024-09-17T00:25:14.321 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:25:14 smithi002 ceph-mon[66732]: pgmap v59: 161 pgs: 38 active+undersized, 20 active+undersized+degraded, 103 active+clean; 585 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 383 B/s rd, 0 op/s; 74/726 objects degraded (10.193%) 2024-09-17T00:25:14.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:25:14 smithi099 ceph-mon[56712]: pgmap v59: 161 pgs: 38 active+undersized, 20 active+undersized+degraded, 103 active+clean; 585 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 383 B/s rd, 0 op/s; 74/726 objects degraded (10.193%) 2024-09-17T00:25:16.213 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:25:16 smithi002 ceph-mon[66732]: pgmap v60: 161 pgs: 38 active+undersized, 20 active+undersized+degraded, 103 active+clean; 585 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 308 B/s rd, 0 op/s; 74/726 objects degraded (10.193%) 2024-09-17T00:25:16.213 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:25:16 smithi002 ceph-mon[66732]: from='osd.0 [v2:172.21.15.2:6802/884190680,v1:172.21.15.2:6803/884190680]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-17T00:25:16.213 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:25:16 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:25:16.213 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:25:16 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:25:16.213 INFO:journalctl@ceph.osd.0.smithi002.stdout:Sep 17 00:25:15 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-0[71240]: 2024-09-17T00:25:15.962+0000 7f9894e32640 -1 osd.0 0 read_superblock omap replica is missing. 2024-09-17T00:25:16.213 INFO:journalctl@ceph.osd.0.smithi002.stdout:Sep 17 00:25:15 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-0[71240]: 2024-09-17T00:25:15.992+0000 7f9894e32640 -1 osd.0 112 log_to_monitors true 2024-09-17T00:25:16.213 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:25:16 smithi002 ceph-mon[63804]: pgmap v60: 161 pgs: 38 active+undersized, 20 active+undersized+degraded, 103 active+clean; 585 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 308 B/s rd, 0 op/s; 74/726 objects degraded (10.193%) 2024-09-17T00:25:16.214 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:25:16 smithi002 ceph-mon[63804]: from='osd.0 [v2:172.21.15.2:6802/884190680,v1:172.21.15.2:6803/884190680]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-17T00:25:16.214 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:25:16 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:25:16.214 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:25:16 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:25:16.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:25:16 smithi099 ceph-mon[56712]: pgmap v60: 161 pgs: 38 active+undersized, 20 active+undersized+degraded, 103 active+clean; 585 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 308 B/s rd, 0 op/s; 74/726 objects degraded (10.193%) 2024-09-17T00:25:16.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:25:16 smithi099 ceph-mon[56712]: from='osd.0 [v2:172.21.15.2:6802/884190680,v1:172.21.15.2:6803/884190680]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-17T00:25:16.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:25:16 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:25:16.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:25:16 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:25:16.479 INFO:journalctl@ceph.osd.0.smithi002.stdout:Sep 17 00:25:16 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-0[71240]: 2024-09-17T00:25:16.215+0000 7f988cbdd640 -1 osd.0 112 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T00:25:16.979 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:25:16 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: ::ffff:172.21.15.99 - - [17/Sep/2024:00:25:16] "GET /metrics HTTP/1.1" 200 38396 "" "Prometheus/2.43.0" 2024-09-17T00:25:17.423 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:25:17 smithi002 ceph-mon[66732]: from='osd.0 [v2:172.21.15.2:6802/884190680,v1:172.21.15.2:6803/884190680]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-17T00:25:17.423 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:25:17 smithi002 ceph-mon[66732]: osdmap e115: 8 total, 7 up, 8 in 2024-09-17T00:25:17.423 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:25:17 smithi002 ceph-mon[66732]: from='osd.0 [v2:172.21.15.2:6802/884190680,v1:172.21.15.2:6803/884190680]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]: dispatch 2024-09-17T00:25:17.424 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:25:17 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:25:17.424 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:25:17 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:25:17.424 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:25:17 smithi002 ceph-mon[63804]: from='osd.0 [v2:172.21.15.2:6802/884190680,v1:172.21.15.2:6803/884190680]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-17T00:25:17.424 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:25:17 smithi002 ceph-mon[63804]: osdmap e115: 8 total, 7 up, 8 in 2024-09-17T00:25:17.424 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:25:17 smithi002 ceph-mon[63804]: from='osd.0 [v2:172.21.15.2:6802/884190680,v1:172.21.15.2:6803/884190680]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]: dispatch 2024-09-17T00:25:17.424 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:25:17 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:25:17.424 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:25:17 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:25:17.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:25:17 smithi099 ceph-mon[56712]: from='osd.0 [v2:172.21.15.2:6802/884190680,v1:172.21.15.2:6803/884190680]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-17T00:25:17.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:25:17 smithi099 ceph-mon[56712]: osdmap e115: 8 total, 7 up, 8 in 2024-09-17T00:25:17.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:25:17 smithi099 ceph-mon[56712]: from='osd.0 [v2:172.21.15.2:6802/884190680,v1:172.21.15.2:6803/884190680]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]: dispatch 2024-09-17T00:25:17.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:25:17 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:25:17.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:25:17 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:25:18.468 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:25:18 smithi099 ceph-mon[56712]: pgmap v62: 161 pgs: 38 active+undersized, 20 active+undersized+degraded, 103 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 307 B/s rd, 0 op/s; 74/726 objects degraded (10.193%) 2024-09-17T00:25:18.468 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:25:18 smithi099 ceph-mon[56712]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-17T00:25:18.468 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:25:18 smithi099 ceph-mon[56712]: osd.0 [v2:172.21.15.2:6802/884190680,v1:172.21.15.2:6803/884190680] boot 2024-09-17T00:25:18.469 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:25:18 smithi099 ceph-mon[56712]: osdmap e116: 8 total, 8 up, 8 in 2024-09-17T00:25:18.469 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:25:18 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T00:25:18.477 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:25:18 smithi002 ceph-mon[66732]: pgmap v62: 161 pgs: 38 active+undersized, 20 active+undersized+degraded, 103 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 307 B/s rd, 0 op/s; 74/726 objects degraded (10.193%) 2024-09-17T00:25:18.477 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:25:18 smithi002 ceph-mon[66732]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-17T00:25:18.478 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:25:18 smithi002 ceph-mon[66732]: osd.0 [v2:172.21.15.2:6802/884190680,v1:172.21.15.2:6803/884190680] boot 2024-09-17T00:25:18.478 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:25:18 smithi002 ceph-mon[66732]: osdmap e116: 8 total, 8 up, 8 in 2024-09-17T00:25:18.478 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:25:18 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T00:25:18.478 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:25:18 smithi002 ceph-mon[63804]: pgmap v62: 161 pgs: 38 active+undersized, 20 active+undersized+degraded, 103 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 307 B/s rd, 0 op/s; 74/726 objects degraded (10.193%) 2024-09-17T00:25:18.478 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:25:18 smithi002 ceph-mon[63804]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-17T00:25:18.478 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:25:18 smithi002 ceph-mon[63804]: osd.0 [v2:172.21.15.2:6802/884190680,v1:172.21.15.2:6803/884190680] boot 2024-09-17T00:25:18.478 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:25:18 smithi002 ceph-mon[63804]: osdmap e116: 8 total, 8 up, 8 in 2024-09-17T00:25:18.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:25:18 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-17T00:25:18.967 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:25:18 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[52757]: ts=2024-09-17T00:25:18.659Z caller=alerting.go:366 level=warn component="rule manager" alert="unsupported value type" msg="Expanding alert template failed" err="error executing template __alert_CephOSDDown: template: __alert_CephOSDDown:1:358: executing \"__alert_CephOSDDown\" at : error calling query: found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.2\", device_class=\"ssd\", hostname=\"smithi002\", instance=\"172.21.15.99:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.2\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"3aad27a2-7489-11ef-bceb-c7b262605968\", cluster_addr=\"172.21.15.2\", device_class=\"ssd\", hostname=\"smithi002\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.2\"}];many-to-many matching not allowed: matching labels must be unique on one side" data="unsupported value type" 2024-09-17T00:25:18.967 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:25:18 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[52757]: ts=2024-09-17T00:25:18.660Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd name=CephOSDFlapping index=13 msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.2\", device_class=\"ssd\", hostname=\"smithi002\", instance=\"172.21.15.99:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.2\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"3aad27a2-7489-11ef-bceb-c7b262605968\", cluster_addr=\"172.21.15.2\", device_class=\"ssd\", hostname=\"smithi002\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.2\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-17T00:25:19.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:25:19 smithi099 ceph-mon[56712]: osdmap e117: 8 total, 8 up, 8 in 2024-09-17T00:25:19.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:25:19 smithi002 ceph-mon[66732]: osdmap e117: 8 total, 8 up, 8 in 2024-09-17T00:25:19.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:25:19 smithi002 ceph-mon[63804]: osdmap e117: 8 total, 8 up, 8 in 2024-09-17T00:25:19.653 INFO:teuthology.orchestra.run.smithi002.stdout:true 2024-09-17T00:25:20.358 INFO:teuthology.orchestra.run.smithi002.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-17T00:25:20.359 INFO:teuthology.orchestra.run.smithi002.stdout:alertmanager.a smithi002 *:9093,9094 running (3m) 4s ago 8m 26.3M - 0.25.0 c8568f914cd2 a21e13b3d677 2024-09-17T00:25:20.359 INFO:teuthology.orchestra.run.smithi002.stdout:grafana.a smithi099 *:3000 running (2m) 53s ago 7m 46.9M - 8.3.5 dad864ee21e9 bafb824a3c08 2024-09-17T00:25:20.359 INFO:teuthology.orchestra.run.smithi002.stdout:iscsi.foo.smithi002.vmouem smithi002 running (21s) 4s ago 6m 50.3M - 3.5 e1d6a67b021e 9d37e0963524 2024-09-17T00:25:20.359 INFO:teuthology.orchestra.run.smithi002.stdout:mgr.x smithi099 *:8443,9283,8765 running (2m) 53s ago 12m 439M - 19.1.1-330-gc2ddf972 ebb0615149de 6aa2ccef9ca0 2024-09-17T00:25:20.359 INFO:teuthology.orchestra.run.smithi002.stdout:mgr.y smithi002 *:8443,9283,8765 running (2m) 4s ago 14m 500M - 19.1.1-330-gc2ddf972 ebb0615149de 8838897e8449 2024-09-17T00:25:20.359 INFO:teuthology.orchestra.run.smithi002.stdout:mon.a smithi002 running (107s) 4s ago 14m 47.3M 2048M 19.1.1-330-gc2ddf972 ebb0615149de a035def53ef6 2024-09-17T00:25:20.360 INFO:teuthology.orchestra.run.smithi002.stdout:mon.b smithi099 running (64s) 53s ago 12m 17.5M 2048M 19.1.1-330-gc2ddf972 ebb0615149de aab3ab73a850 2024-09-17T00:25:20.360 INFO:teuthology.orchestra.run.smithi002.stdout:mon.c smithi002 running (81s) 4s ago 13m 38.9M 2048M 19.1.1-330-gc2ddf972 ebb0615149de 8a510255ae66 2024-09-17T00:25:20.360 INFO:teuthology.orchestra.run.smithi002.stdout:node-exporter.a smithi002 *:9100 running (3m) 4s ago 8m 14.7M - 1.5.0 0da6a335fe13 1030ae01bb32 2024-09-17T00:25:20.360 INFO:teuthology.orchestra.run.smithi002.stdout:node-exporter.b smithi099 *:9100 running (2m) 53s ago 8m 16.8M - 1.5.0 0da6a335fe13 01fcd35dca8a 2024-09-17T00:25:20.360 INFO:teuthology.orchestra.run.smithi002.stdout:osd.0 smithi002 running (9s) 4s ago 12m 12.1M 2737M 19.1.1-330-gc2ddf972 ebb0615149de 8345719cb6f0 2024-09-17T00:25:20.360 INFO:teuthology.orchestra.run.smithi002.stdout:osd.1 smithi002 running (11m) 4s ago 11m 56.4M 2737M 17.2.0 e1d6a67b021e 7cb62c311c7d 2024-09-17T00:25:20.360 INFO:teuthology.orchestra.run.smithi002.stdout:osd.2 smithi002 running (11m) 4s ago 11m 53.3M 2737M 17.2.0 e1d6a67b021e eafddacb955e 2024-09-17T00:25:20.360 INFO:teuthology.orchestra.run.smithi002.stdout:osd.3 smithi002 running (11m) 4s ago 11m 57.0M 2737M 17.2.0 e1d6a67b021e fd99d3764412 2024-09-17T00:25:20.361 INFO:teuthology.orchestra.run.smithi002.stdout:osd.4 smithi099 running (10m) 53s ago 10m 55.4M 2992M 17.2.0 e1d6a67b021e 0af1827b658d 2024-09-17T00:25:20.361 INFO:teuthology.orchestra.run.smithi002.stdout:osd.5 smithi099 running (10m) 53s ago 10m 54.2M 2992M 17.2.0 e1d6a67b021e 4f7ef3aefc58 2024-09-17T00:25:20.361 INFO:teuthology.orchestra.run.smithi002.stdout:osd.6 smithi099 running (10m) 53s ago 9m 51.0M 2992M 17.2.0 e1d6a67b021e 3213f5e6952a 2024-09-17T00:25:20.361 INFO:teuthology.orchestra.run.smithi002.stdout:osd.7 smithi099 running (9m) 53s ago 9m 56.6M 2992M 17.2.0 e1d6a67b021e 63be3a1ed0a1 2024-09-17T00:25:20.361 INFO:teuthology.orchestra.run.smithi002.stdout:prometheus.a smithi099 *:9095 running (2m) 53s ago 8m 50.1M - 2.43.0 a07b618ecd1d bf32c2ae11dc 2024-09-17T00:25:20.361 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.foo.smithi002.uwdhcs smithi002 *:8000 running (7m) 4s ago 7m 75.6M - 17.2.0 e1d6a67b021e 9c35813f459a 2024-09-17T00:25:20.361 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.foo.smithi099.utzfap smithi099 *:8000 running (7m) 53s ago 7m 76.0M - 17.2.0 e1d6a67b021e 34ec989160a6 2024-09-17T00:25:20.362 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.smpl.smithi002.rqnxii smithi002 *:80 running (7m) 4s ago 7m 74.0M - 17.2.0 e1d6a67b021e 0e180d0afb99 2024-09-17T00:25:20.362 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.smpl.smithi099.nzvkpm smithi099 *:80 running (7m) 53s ago 7m 76.5M - 17.2.0 e1d6a67b021e 6a068e82ac72 2024-09-17T00:25:20.362 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:25:20 smithi002 ceph-mon[66732]: pgmap v65: 161 pgs: 37 peering, 14 active+undersized, 7 active+undersized+degraded, 103 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 32/726 objects degraded (4.408%) 2024-09-17T00:25:20.362 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:25:20 smithi002 ceph-mon[66732]: from='client.44266 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:25:20.362 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:25:20 smithi002 ceph-mon[66732]: Health check update: Degraded data redundancy: 32/726 objects degraded (4.408%), 7 pgs degraded (PG_DEGRADED) 2024-09-17T00:25:20.362 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:25:20 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:25:20.363 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:25:20 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:25:20.363 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:25:20 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:25:20.363 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:25:20 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:25:20.363 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:25:20 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:25:20.363 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:25:20 smithi002 ceph-mon[63804]: pgmap v65: 161 pgs: 37 peering, 14 active+undersized, 7 active+undersized+degraded, 103 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 32/726 objects degraded (4.408%) 2024-09-17T00:25:20.363 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:25:20 smithi002 ceph-mon[63804]: from='client.44266 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:25:20.363 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:25:20 smithi002 ceph-mon[63804]: Health check update: Degraded data redundancy: 32/726 objects degraded (4.408%), 7 pgs degraded (PG_DEGRADED) 2024-09-17T00:25:20.363 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:25:20 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:25:20.363 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:25:20 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:25:20.363 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:25:20 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:25:20.364 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:25:20 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:25:20.364 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:25:20 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:25:20.468 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:25:20 smithi099 ceph-mon[56712]: pgmap v65: 161 pgs: 37 peering, 14 active+undersized, 7 active+undersized+degraded, 103 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 32/726 objects degraded (4.408%) 2024-09-17T00:25:20.468 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:25:20 smithi099 ceph-mon[56712]: from='client.44266 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:25:20.468 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:25:20 smithi099 ceph-mon[56712]: Health check update: Degraded data redundancy: 32/726 objects degraded (4.408%), 7 pgs degraded (PG_DEGRADED) 2024-09-17T00:25:20.468 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:25:20 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:25:20.468 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:25:20 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:25:20.469 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:25:20 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:25:20.469 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:25:20 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:25:20.469 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:25:20 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:25:20.802 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-09-17T00:25:20.802 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": { 2024-09-17T00:25:20.803 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 19.1.1-330-gc2ddf972 (c2ddf9721ef254645201cdbabd9c8f49c30a586f) squid (stable)": 3 2024-09-17T00:25:20.803 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-09-17T00:25:20.803 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": { 2024-09-17T00:25:20.803 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 19.1.1-330-gc2ddf972 (c2ddf9721ef254645201cdbabd9c8f49c30a586f) squid (stable)": 2 2024-09-17T00:25:20.803 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-09-17T00:25:20.803 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": { 2024-09-17T00:25:20.803 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 7, 2024-09-17T00:25:20.804 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 19.1.1-330-gc2ddf972 (c2ddf9721ef254645201cdbabd9c8f49c30a586f) squid (stable)": 1 2024-09-17T00:25:20.804 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-09-17T00:25:20.804 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw": { 2024-09-17T00:25:20.804 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-09-17T00:25:20.804 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-09-17T00:25:20.804 INFO:teuthology.orchestra.run.smithi002.stdout: "overall": { 2024-09-17T00:25:20.804 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 11, 2024-09-17T00:25:20.804 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 19.1.1-330-gc2ddf972 (c2ddf9721ef254645201cdbabd9c8f49c30a586f) squid (stable)": 6 2024-09-17T00:25:20.805 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-09-17T00:25:20.805 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-09-17T00:25:21.159 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-09-17T00:25:21.159 INFO:teuthology.orchestra.run.smithi002.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f", 2024-09-17T00:25:21.159 INFO:teuthology.orchestra.run.smithi002.stdout: "in_progress": true, 2024-09-17T00:25:21.159 INFO:teuthology.orchestra.run.smithi002.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-17T00:25:21.159 INFO:teuthology.orchestra.run.smithi002.stdout: "services_complete": [ 2024-09-17T00:25:21.160 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr", 2024-09-17T00:25:21.160 INFO:teuthology.orchestra.run.smithi002.stdout: "mon" 2024-09-17T00:25:21.160 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-09-17T00:25:21.160 INFO:teuthology.orchestra.run.smithi002.stdout: "progress": "6/23 daemons upgraded", 2024-09-17T00:25:21.160 INFO:teuthology.orchestra.run.smithi002.stdout: "message": "Currently upgrading osd daemons", 2024-09-17T00:25:21.160 INFO:teuthology.orchestra.run.smithi002.stdout: "is_paused": false 2024-09-17T00:25:21.160 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-09-17T00:25:21.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:25:21 smithi099 ceph-mon[56712]: from='client.34380 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:25:21.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:25:21 smithi099 ceph-mon[56712]: from='client.34386 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:25:21.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:25:21 smithi099 ceph-mon[56712]: pgmap v66: 161 pgs: 37 peering, 14 active+undersized, 7 active+undersized+degraded, 103 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 971 B/s rd, 0 op/s; 32/726 objects degraded (4.408%) 2024-09-17T00:25:21.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:25:21 smithi099 ceph-mon[56712]: Reconfiguring iscsi.foo.smithi002.vmouem (dependencies changed)... 2024-09-17T00:25:21.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:25:21 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi002.vmouem", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-17T00:25:21.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:25:21 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi002.vmouem", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-17T00:25:21.468 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:25:21 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:25:21.468 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:25:21 smithi099 ceph-mon[56712]: Reconfiguring daemon iscsi.foo.smithi002.vmouem on smithi002 2024-09-17T00:25:21.468 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:25:21 smithi099 ceph-mon[56712]: from='client.44299 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:25:21.468 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:25:21 smithi099 ceph-mon[56712]: from='client.? 172.21.15.2:0/1511172421' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:25:21.468 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:25:21 smithi099 ceph-mon[56712]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-09-17T00:25:21.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:25:21 smithi002 ceph-mon[66732]: from='client.34380 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:25:21.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:25:21 smithi002 ceph-mon[66732]: from='client.34386 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:25:21.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:25:21 smithi002 ceph-mon[66732]: pgmap v66: 161 pgs: 37 peering, 14 active+undersized, 7 active+undersized+degraded, 103 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 971 B/s rd, 0 op/s; 32/726 objects degraded (4.408%) 2024-09-17T00:25:21.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:25:21 smithi002 ceph-mon[66732]: Reconfiguring iscsi.foo.smithi002.vmouem (dependencies changed)... 2024-09-17T00:25:21.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:25:21 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi002.vmouem", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-17T00:25:21.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:25:21 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi002.vmouem", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-17T00:25:21.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:25:21 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:25:21.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:25:21 smithi002 ceph-mon[66732]: Reconfiguring daemon iscsi.foo.smithi002.vmouem on smithi002 2024-09-17T00:25:21.480 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:25:21 smithi002 ceph-mon[66732]: from='client.44299 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:25:21.480 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:25:21 smithi002 ceph-mon[66732]: from='client.? 172.21.15.2:0/1511172421' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:25:21.480 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:25:21 smithi002 ceph-mon[66732]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-09-17T00:25:21.480 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:25:21 smithi002 ceph-mon[63804]: from='client.34380 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:25:21.480 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:25:21 smithi002 ceph-mon[63804]: from='client.34386 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:25:21.480 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:25:21 smithi002 ceph-mon[63804]: pgmap v66: 161 pgs: 37 peering, 14 active+undersized, 7 active+undersized+degraded, 103 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 971 B/s rd, 0 op/s; 32/726 objects degraded (4.408%) 2024-09-17T00:25:21.480 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:25:21 smithi002 ceph-mon[63804]: Reconfiguring iscsi.foo.smithi002.vmouem (dependencies changed)... 2024-09-17T00:25:21.480 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:25:21 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi002.vmouem", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-17T00:25:21.480 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:25:21 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi002.vmouem", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-17T00:25:21.480 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:25:21 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:25:21.480 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:25:21 smithi002 ceph-mon[63804]: Reconfiguring daemon iscsi.foo.smithi002.vmouem on smithi002 2024-09-17T00:25:21.481 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:25:21 smithi002 ceph-mon[63804]: from='client.44299 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:25:21.481 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:25:21 smithi002 ceph-mon[63804]: from='client.? 172.21.15.2:0/1511172421' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:25:21.481 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:25:21 smithi002 ceph-mon[63804]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-09-17T00:25:21.577 INFO:teuthology.orchestra.run.smithi002.stdout:HEALTH_WARN Degraded data redundancy: 32/726 objects degraded (4.408%), 7 pgs degraded 2024-09-17T00:25:21.578 INFO:teuthology.orchestra.run.smithi002.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 32/726 objects degraded (4.408%), 7 pgs degraded 2024-09-17T00:25:21.578 INFO:teuthology.orchestra.run.smithi002.stdout: pg 3.1 is active+undersized+degraded, acting [4,3] 2024-09-17T00:25:21.578 INFO:teuthology.orchestra.run.smithi002.stdout: pg 3.6 is active+undersized+degraded, acting [1,4] 2024-09-17T00:25:21.578 INFO:teuthology.orchestra.run.smithi002.stdout: pg 3.12 is active+undersized+degraded, acting [7,3] 2024-09-17T00:25:21.578 INFO:teuthology.orchestra.run.smithi002.stdout: pg 3.17 is active+undersized+degraded, acting [5,3] 2024-09-17T00:25:21.578 INFO:teuthology.orchestra.run.smithi002.stdout: pg 3.1b is active+undersized+degraded, acting [4,7] 2024-09-17T00:25:21.578 INFO:teuthology.orchestra.run.smithi002.stdout: pg 3.1f is active+undersized+degraded, acting [5,2] 2024-09-17T00:25:21.578 INFO:teuthology.orchestra.run.smithi002.stdout: pg 4.3 is active+undersized+degraded, acting [5,7] 2024-09-17T00:25:22.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:25:22 smithi099 ceph-mon[56712]: from='client.34404 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:25:22.540 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:25:22 smithi099 ceph-mon[56712]: from='client.? 172.21.15.2:0/746994055' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T00:25:22.540 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:25:22 smithi002 ceph-mon[63804]: from='client.34404 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:25:22.540 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:25:22 smithi002 ceph-mon[63804]: from='client.? 172.21.15.2:0/746994055' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T00:25:22.541 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:25:22 smithi002 ceph-mon[66732]: from='client.34404 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:25:22.541 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:25:22 smithi002 ceph-mon[66732]: from='client.? 172.21.15.2:0/746994055' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T00:25:23.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:25:23 smithi099 ceph-mon[56712]: pgmap v67: 161 pgs: 37 peering, 124 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 859 B/s rd, 0 op/s 2024-09-17T00:25:23.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:25:23 smithi099 ceph-mon[56712]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 32/726 objects degraded (4.408%), 7 pgs degraded) 2024-09-17T00:25:23.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:25:23 smithi099 ceph-mon[56712]: Cluster is now healthy 2024-09-17T00:25:23.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:25:23 smithi002 ceph-mon[66732]: pgmap v67: 161 pgs: 37 peering, 124 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 859 B/s rd, 0 op/s 2024-09-17T00:25:23.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:25:23 smithi002 ceph-mon[66732]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 32/726 objects degraded (4.408%), 7 pgs degraded) 2024-09-17T00:25:23.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:25:23 smithi002 ceph-mon[66732]: Cluster is now healthy 2024-09-17T00:25:23.480 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:25:23 smithi002 ceph-mon[63804]: pgmap v67: 161 pgs: 37 peering, 124 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 859 B/s rd, 0 op/s 2024-09-17T00:25:23.480 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:25:23 smithi002 ceph-mon[63804]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 32/726 objects degraded (4.408%), 7 pgs degraded) 2024-09-17T00:25:23.480 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:25:23 smithi002 ceph-mon[63804]: Cluster is now healthy 2024-09-17T00:25:23.967 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:25:23 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[52757]: ts=2024-09-17T00:25:23.506Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes name=CephNodeDiskspaceWarning index=4 msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi002\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi002\", job=\"node\", machine=\"x86_64\", nodename=\"smithi002\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"3aad27a2-7489-11ef-bceb-c7b262605968\", domainname=\"(none)\", instance=\"smithi002\", job=\"node\", machine=\"x86_64\", nodename=\"smithi002\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-17T00:25:23.979 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:25:23 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: 2024-09-17T00:25:23.616+0000 7f17e1e2b640 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi002.vmouem ... 2024-09-17T00:25:23.980 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:25:23 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: Creating ceph-iscsi config... 2024-09-17T00:25:23.980 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:25:23 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/iscsi-gateway.cfg 2024-09-17T00:25:23.980 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:25:23 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/tcmu-runner-entrypoint.sh 2024-09-17T00:25:23.980 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:25:23 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: Non-zero exit code 1 from systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem 2024-09-17T00:25:23.980 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:25:23 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: systemctl: stderr Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:25:23.980 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:25:23 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: systemctl: stderr See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:25:23.980 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:25:23 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: Traceback (most recent call last): 2024-09-17T00:25:23.980 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:25:23 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-17T00:25:23.980 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:25:23 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: return _run_code(code, main_globals, None, 2024-09-17T00:25:23.981 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:25:23 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-17T00:25:23.981 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:25:23 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: exec(code, run_globals) 2024-09-17T00:25:23.981 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:25:23 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5579, in 2024-09-17T00:25:23.981 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:25:23 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5567, in main 2024-09-17T00:25:23.981 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:25:23 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3049, in command_deploy_from 2024-09-17T00:25:23.981 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:25:23 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3084, in _common_deploy 2024-09-17T00:25:23.981 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:25:23 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3104, in _deploy_daemon_container 2024-09-17T00:25:23.981 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:25:23 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 1124, in deploy_daemon 2024-09-17T00:25:23.981 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:25:23 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/tmp/tmpnroy8d0w.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-17T00:25:23.982 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:25:23 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: RuntimeError: Failed command: systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem: Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:25:23.982 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:25:23 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:25:23.982 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:25:23 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: Traceback (most recent call last): 2024-09-17T00:25:23.983 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:25:23 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1134, in _check_daemons 2024-09-17T00:25:23.983 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:25:23 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: self.mgr._daemon_action(daemon_spec, action=action) 2024-09-17T00:25:23.983 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:25:23 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2473, in _daemon_action 2024-09-17T00:25:23.983 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:25:23 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: return self.wait_async( 2024-09-17T00:25:23.983 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:25:23 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/usr/share/ceph/mgr/cephadm/module.py", line 796, in wait_async 2024-09-17T00:25:23.983 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:25:23 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: return self.event_loop.get_result(coro, timeout) 2024-09-17T00:25:23.983 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:25:23 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-09-17T00:25:23.983 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:25:23 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: return future.result(timeout) 2024-09-17T00:25:23.983 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:25:23 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/lib64/python3.9/concurrent/futures/_base.py", line 446, in result 2024-09-17T00:25:23.983 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:25:23 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: return self.__get_result() 2024-09-17T00:25:23.983 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:25:23 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/lib64/python3.9/concurrent/futures/_base.py", line 391, in __get_result 2024-09-17T00:25:23.983 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:25:23 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: raise self._exception 2024-09-17T00:25:23.984 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:25:23 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1370, in _create_daemon 2024-09-17T00:25:23.984 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:25:23 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: out, err, code = await self._run_cephadm( 2024-09-17T00:25:23.984 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:25:23 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1713, in _run_cephadm 2024-09-17T00:25:23.984 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:25:23 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: raise OrchestratorError( 2024-09-17T00:25:23.984 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:25:23 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi002.vmouem ... 2024-09-17T00:25:23.984 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:25:23 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: Creating ceph-iscsi config... 2024-09-17T00:25:23.984 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:25:23 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/iscsi-gateway.cfg 2024-09-17T00:25:23.984 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:25:23 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/tcmu-runner-entrypoint.sh 2024-09-17T00:25:23.984 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:25:23 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: Non-zero exit code 1 from systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem 2024-09-17T00:25:23.984 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:25:23 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: systemctl: stderr Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:25:23.984 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:25:23 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: systemctl: stderr See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:25:23.984 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:25:23 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: Traceback (most recent call last): 2024-09-17T00:25:23.984 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:25:23 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-17T00:25:23.985 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:25:23 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: return _run_code(code, main_globals, None, 2024-09-17T00:25:23.985 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:25:23 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-17T00:25:23.985 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:25:23 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: exec(code, run_globals) 2024-09-17T00:25:23.985 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:25:23 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5579, in 2024-09-17T00:25:23.985 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:25:23 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5567, in main 2024-09-17T00:25:23.985 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:25:23 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3049, in command_deploy_from 2024-09-17T00:25:23.985 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:25:23 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3084, in _common_deploy 2024-09-17T00:25:23.985 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:25:23 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3104, in _deploy_daemon_container 2024-09-17T00:25:23.985 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:25:23 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 1124, in deploy_daemon 2024-09-17T00:25:23.985 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:25:23 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/tmp/tmpnroy8d0w.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-17T00:25:23.986 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:25:23 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: RuntimeError: Failed command: systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem: Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:25:23.986 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:25:23 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:25:23.986 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:25:23 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: 2024-09-17T00:25:23.660+0000 7f17efec7640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (14 PGs are or would become offline) 2024-09-17T00:25:24.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:25:24 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T00:25:24.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:25:24 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:25:24.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:25:24 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:25:24.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:25:24 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:25:24.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:25:24 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-09-17T00:25:24.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:25:24 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T00:25:24.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:25:24 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:25:24.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:25:24 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:25:24.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:25:24 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:25:24.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:25:24 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-09-17T00:25:24.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:25:24 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T00:25:24.480 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:25:24 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:25:24.480 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:25:24 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:25:24.480 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:25:24 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:25:24.480 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:25:24 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-09-17T00:25:25.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:25:25 smithi099 ceph-mon[56712]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi002.vmouem ... 2024-09-17T00:25:25.467 INFO:journalctl@ceph.mon.b.smithi099.stdout: Creating ceph-iscsi config... 2024-09-17T00:25:25.468 INFO:journalctl@ceph.mon.b.smithi099.stdout: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/iscsi-gateway.cfg 2024-09-17T00:25:25.468 INFO:journalctl@ceph.mon.b.smithi099.stdout: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/tcmu-runner-entrypoint.sh 2024-09-17T00:25:25.468 INFO:journalctl@ceph.mon.b.smithi099.stdout: Non-zero exit code 1 from systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem 2024-09-17T00:25:25.468 INFO:journalctl@ceph.mon.b.smithi099.stdout: systemctl: stderr Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:25:25.468 INFO:journalctl@ceph.mon.b.smithi099.stdout: systemctl: stderr See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:25:25.468 INFO:journalctl@ceph.mon.b.smithi099.stdout: Traceback (most recent call last): 2024-09-17T00:25:25.469 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-17T00:25:25.469 INFO:journalctl@ceph.mon.b.smithi099.stdout: return _run_code(code, main_globals, None, 2024-09-17T00:25:25.469 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-17T00:25:25.469 INFO:journalctl@ceph.mon.b.smithi099.stdout: exec(code, run_globals) 2024-09-17T00:25:25.469 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5579, in 2024-09-17T00:25:25.469 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5567, in main 2024-09-17T00:25:25.470 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3049, in command_deploy_from 2024-09-17T00:25:25.470 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3084, in _common_deploy 2024-09-17T00:25:25.470 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3104, in _deploy_daemon_container 2024-09-17T00:25:25.470 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 1124, in deploy_daemon 2024-09-17T00:25:25.470 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-17T00:25:25.470 INFO:journalctl@ceph.mon.b.smithi099.stdout: RuntimeError: Failed command: systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem: Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:25:25.471 INFO:journalctl@ceph.mon.b.smithi099.stdout: See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:25:25.471 INFO:journalctl@ceph.mon.b.smithi099.stdout: Traceback (most recent call last): 2024-09-17T00:25:25.471 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1134, in _check_daemons 2024-09-17T00:25:25.471 INFO:journalctl@ceph.mon.b.smithi099.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-09-17T00:25:25.471 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2473, in _daemon_action 2024-09-17T00:25:25.471 INFO:journalctl@ceph.mon.b.smithi099.stdout: return self.wait_async( 2024-09-17T00:25:25.472 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 796, in wait_async 2024-09-17T00:25:25.472 INFO:journalctl@ceph.mon.b.smithi099.stdout: return self.event_loop.get_result(coro, timeout) 2024-09-17T00:25:25.472 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-09-17T00:25:25.472 INFO:journalctl@ceph.mon.b.smithi099.stdout: return future.result(timeout) 2024-09-17T00:25:25.472 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/lib64/python3.9/concurrent/futures/_base.py", line 446, in result 2024-09-17T00:25:25.472 INFO:journalctl@ceph.mon.b.smithi099.stdout: return self.__get_result() 2024-09-17T00:25:25.473 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/lib64/python3.9/concurrent/futures/_base.py", line 391, in __get_result 2024-09-17T00:25:25.473 INFO:journalctl@ceph.mon.b.smithi099.stdout: raise self._exception 2024-09-17T00:25:25.473 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1370, in _create_daemon 2024-09-17T00:25:25.473 INFO:journalctl@ceph.mon.b.smithi099.stdout: out, err, code = await self._run_cephadm( 2024-09-17T00:25:25.473 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1713, in _run_cephadm 2024-09-17T00:25:25.473 INFO:journalctl@ceph.mon.b.smithi099.stdout: raise OrchestratorError( 2024-09-17T00:25:25.474 INFO:journalctl@ceph.mon.b.smithi099.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi002.vmouem ... 2024-09-17T00:25:25.474 INFO:journalctl@ceph.mon.b.smithi099.stdout: Creating ceph-iscsi config... 2024-09-17T00:25:25.474 INFO:journalctl@ceph.mon.b.smithi099.stdout: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/iscsi-gateway.cfg 2024-09-17T00:25:25.474 INFO:journalctl@ceph.mon.b.smithi099.stdout: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/tcmu-runner-entrypoint.sh 2024-09-17T00:25:25.474 INFO:journalctl@ceph.mon.b.smithi099.stdout: Non-zero exit code 1 from systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem 2024-09-17T00:25:25.474 INFO:journalctl@ceph.mon.b.smithi099.stdout: systemctl: stderr Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:25:25.475 INFO:journalctl@ceph.mon.b.smithi099.stdout: systemctl: stderr See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:25:25.475 INFO:journalctl@ceph.mon.b.smithi099.stdout: Traceback (most recent call last): 2024-09-17T00:25:25.475 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-17T00:25:25.475 INFO:journalctl@ceph.mon.b.smithi099.stdout: return _run_code(code, main_globals, None, 2024-09-17T00:25:25.475 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-17T00:25:25.475 INFO:journalctl@ceph.mon.b.smithi099.stdout: exec(code, run_globals) 2024-09-17T00:25:25.476 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5579, in 2024-09-17T00:25:25.476 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5567, in main 2024-09-17T00:25:25.476 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3049, in command_deploy_from 2024-09-17T00:25:25.476 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3084, in _common_deploy 2024-09-17T00:25:25.476 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3104, in _deploy_daemon_container 2024-09-17T00:25:25.476 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 1124, in deploy_daemon 2024-09-17T00:25:25.477 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-17T00:25:25.477 INFO:journalctl@ceph.mon.b.smithi099.stdout: RuntimeError: Failed command: systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem: Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:25:25.477 INFO:journalctl@ceph.mon.b.smithi099.stdout: See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:25:25.477 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:25:25 smithi099 ceph-mon[56712]: from='mon.1 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-09-17T00:25:25.477 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:25:25 smithi099 ceph-mon[56712]: Upgrade: unsafe to stop osd(s) at this time (14 PGs are or would become offline) 2024-09-17T00:25:25.478 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:25:25 smithi099 ceph-mon[56712]: pgmap v68: 161 pgs: 161 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-09-17T00:25:25.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:25:25 smithi002 ceph-mon[63804]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi002.vmouem ... 2024-09-17T00:25:25.479 INFO:journalctl@ceph.mon.a.smithi002.stdout: Creating ceph-iscsi config... 2024-09-17T00:25:25.480 INFO:journalctl@ceph.mon.a.smithi002.stdout: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/iscsi-gateway.cfg 2024-09-17T00:25:25.480 INFO:journalctl@ceph.mon.a.smithi002.stdout: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/tcmu-runner-entrypoint.sh 2024-09-17T00:25:25.480 INFO:journalctl@ceph.mon.a.smithi002.stdout: Non-zero exit code 1 from systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem 2024-09-17T00:25:25.480 INFO:journalctl@ceph.mon.a.smithi002.stdout: systemctl: stderr Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:25:25.480 INFO:journalctl@ceph.mon.a.smithi002.stdout: systemctl: stderr See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:25:25.481 INFO:journalctl@ceph.mon.a.smithi002.stdout: Traceback (most recent call last): 2024-09-17T00:25:25.481 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-17T00:25:25.481 INFO:journalctl@ceph.mon.a.smithi002.stdout: return _run_code(code, main_globals, None, 2024-09-17T00:25:25.481 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-17T00:25:25.481 INFO:journalctl@ceph.mon.a.smithi002.stdout: exec(code, run_globals) 2024-09-17T00:25:25.481 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5579, in 2024-09-17T00:25:25.482 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5567, in main 2024-09-17T00:25:25.482 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3049, in command_deploy_from 2024-09-17T00:25:25.482 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3084, in _common_deploy 2024-09-17T00:25:25.482 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3104, in _deploy_daemon_container 2024-09-17T00:25:25.483 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 1124, in deploy_daemon 2024-09-17T00:25:25.483 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-17T00:25:25.483 INFO:journalctl@ceph.mon.a.smithi002.stdout: RuntimeError: Failed command: systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem: Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:25:25.483 INFO:journalctl@ceph.mon.a.smithi002.stdout: See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:25:25.483 INFO:journalctl@ceph.mon.a.smithi002.stdout: Traceback (most recent call last): 2024-09-17T00:25:25.484 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1134, in _check_daemons 2024-09-17T00:25:25.484 INFO:journalctl@ceph.mon.a.smithi002.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-09-17T00:25:25.484 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2473, in _daemon_action 2024-09-17T00:25:25.484 INFO:journalctl@ceph.mon.a.smithi002.stdout: return self.wait_async( 2024-09-17T00:25:25.484 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 796, in wait_async 2024-09-17T00:25:25.485 INFO:journalctl@ceph.mon.a.smithi002.stdout: return self.event_loop.get_result(coro, timeout) 2024-09-17T00:25:25.485 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-09-17T00:25:25.485 INFO:journalctl@ceph.mon.a.smithi002.stdout: return future.result(timeout) 2024-09-17T00:25:25.485 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/lib64/python3.9/concurrent/futures/_base.py", line 446, in result 2024-09-17T00:25:25.486 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:25:25 smithi002 ceph-mon[66732]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi002.vmouem ... 2024-09-17T00:25:25.486 INFO:journalctl@ceph.mon.c.smithi002.stdout: Creating ceph-iscsi config... 2024-09-17T00:25:25.487 INFO:journalctl@ceph.mon.c.smithi002.stdout: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/iscsi-gateway.cfg 2024-09-17T00:25:25.487 INFO:journalctl@ceph.mon.c.smithi002.stdout: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/tcmu-runner-entrypoint.sh 2024-09-17T00:25:25.487 INFO:journalctl@ceph.mon.c.smithi002.stdout: Non-zero exit code 1 from systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem 2024-09-17T00:25:25.487 INFO:journalctl@ceph.mon.c.smithi002.stdout: systemctl: stderr Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:25:25.487 INFO:journalctl@ceph.mon.c.smithi002.stdout: systemctl: stderr See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:25:25.488 INFO:journalctl@ceph.mon.c.smithi002.stdout: Traceback (most recent call last): 2024-09-17T00:25:25.488 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-17T00:25:25.488 INFO:journalctl@ceph.mon.c.smithi002.stdout: return _run_code(code, main_globals, None, 2024-09-17T00:25:25.488 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-17T00:25:25.489 INFO:journalctl@ceph.mon.c.smithi002.stdout: exec(code, run_globals) 2024-09-17T00:25:25.489 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5579, in 2024-09-17T00:25:25.489 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5567, in main 2024-09-17T00:25:25.489 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3049, in command_deploy_from 2024-09-17T00:25:25.489 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3084, in _common_deploy 2024-09-17T00:25:25.490 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3104, in _deploy_daemon_container 2024-09-17T00:25:25.490 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 1124, in deploy_daemon 2024-09-17T00:25:25.490 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-17T00:25:25.490 INFO:journalctl@ceph.mon.c.smithi002.stdout: RuntimeError: Failed command: systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem: Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:25:25.491 INFO:journalctl@ceph.mon.c.smithi002.stdout: See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:25:25.491 INFO:journalctl@ceph.mon.c.smithi002.stdout: Traceback (most recent call last): 2024-09-17T00:25:25.491 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1134, in _check_daemons 2024-09-17T00:25:25.491 INFO:journalctl@ceph.mon.c.smithi002.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-09-17T00:25:25.491 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2473, in _daemon_action 2024-09-17T00:25:25.492 INFO:journalctl@ceph.mon.c.smithi002.stdout: return self.wait_async( 2024-09-17T00:25:25.492 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 796, in wait_async 2024-09-17T00:25:25.492 INFO:journalctl@ceph.mon.c.smithi002.stdout: return self.event_loop.get_result(coro, timeout) 2024-09-17T00:25:25.492 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-09-17T00:25:25.492 INFO:journalctl@ceph.mon.c.smithi002.stdout: return future.result(timeout) 2024-09-17T00:25:25.493 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/lib64/python3.9/concurrent/futures/_base.py", line 446, in result 2024-09-17T00:25:25.493 INFO:journalctl@ceph.mon.a.smithi002.stdout: return self.__get_result() 2024-09-17T00:25:25.494 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/lib64/python3.9/concurrent/futures/_base.py", line 391, in __get_result 2024-09-17T00:25:25.494 INFO:journalctl@ceph.mon.a.smithi002.stdout: raise self._exception 2024-09-17T00:25:25.494 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1370, in _create_daemon 2024-09-17T00:25:25.494 INFO:journalctl@ceph.mon.a.smithi002.stdout: out, err, code = await self._run_cephadm( 2024-09-17T00:25:25.494 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1713, in _run_cephadm 2024-09-17T00:25:25.495 INFO:journalctl@ceph.mon.a.smithi002.stdout: raise OrchestratorError( 2024-09-17T00:25:25.495 INFO:journalctl@ceph.mon.a.smithi002.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi002.vmouem ... 2024-09-17T00:25:25.495 INFO:journalctl@ceph.mon.a.smithi002.stdout: Creating ceph-iscsi config... 2024-09-17T00:25:25.495 INFO:journalctl@ceph.mon.a.smithi002.stdout: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/iscsi-gateway.cfg 2024-09-17T00:25:25.496 INFO:journalctl@ceph.mon.a.smithi002.stdout: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/tcmu-runner-entrypoint.sh 2024-09-17T00:25:25.496 INFO:journalctl@ceph.mon.a.smithi002.stdout: Non-zero exit code 1 from systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem 2024-09-17T00:25:25.496 INFO:journalctl@ceph.mon.a.smithi002.stdout: systemctl: stderr Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:25:25.496 INFO:journalctl@ceph.mon.a.smithi002.stdout: systemctl: stderr See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:25:25.496 INFO:journalctl@ceph.mon.a.smithi002.stdout: Traceback (most recent call last): 2024-09-17T00:25:25.497 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-17T00:25:25.497 INFO:journalctl@ceph.mon.a.smithi002.stdout: return _run_code(code, main_globals, None, 2024-09-17T00:25:25.497 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-17T00:25:25.497 INFO:journalctl@ceph.mon.a.smithi002.stdout: exec(code, run_globals) 2024-09-17T00:25:25.497 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5579, in 2024-09-17T00:25:25.498 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5567, in main 2024-09-17T00:25:25.498 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3049, in command_deploy_from 2024-09-17T00:25:25.498 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3084, in _common_deploy 2024-09-17T00:25:25.498 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3104, in _deploy_daemon_container 2024-09-17T00:25:25.499 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 1124, in deploy_daemon 2024-09-17T00:25:25.499 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-17T00:25:25.499 INFO:journalctl@ceph.mon.a.smithi002.stdout: RuntimeError: Failed command: systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem: Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:25:25.499 INFO:journalctl@ceph.mon.a.smithi002.stdout: See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:25:25.499 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:25:25 smithi002 ceph-mon[63804]: from='mon.1 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-09-17T00:25:25.500 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:25:25 smithi002 ceph-mon[63804]: Upgrade: unsafe to stop osd(s) at this time (14 PGs are or would become offline) 2024-09-17T00:25:25.500 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:25:25 smithi002 ceph-mon[63804]: pgmap v68: 161 pgs: 161 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-09-17T00:25:25.500 INFO:journalctl@ceph.mon.c.smithi002.stdout: return self.__get_result() 2024-09-17T00:25:25.501 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/lib64/python3.9/concurrent/futures/_base.py", line 391, in __get_result 2024-09-17T00:25:25.501 INFO:journalctl@ceph.mon.c.smithi002.stdout: raise self._exception 2024-09-17T00:25:25.501 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1370, in _create_daemon 2024-09-17T00:25:25.501 INFO:journalctl@ceph.mon.c.smithi002.stdout: out, err, code = await self._run_cephadm( 2024-09-17T00:25:25.501 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1713, in _run_cephadm 2024-09-17T00:25:25.502 INFO:journalctl@ceph.mon.c.smithi002.stdout: raise OrchestratorError( 2024-09-17T00:25:25.502 INFO:journalctl@ceph.mon.c.smithi002.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi002.vmouem ... 2024-09-17T00:25:25.502 INFO:journalctl@ceph.mon.c.smithi002.stdout: Creating ceph-iscsi config... 2024-09-17T00:25:25.502 INFO:journalctl@ceph.mon.c.smithi002.stdout: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/iscsi-gateway.cfg 2024-09-17T00:25:25.502 INFO:journalctl@ceph.mon.c.smithi002.stdout: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/tcmu-runner-entrypoint.sh 2024-09-17T00:25:25.503 INFO:journalctl@ceph.mon.c.smithi002.stdout: Non-zero exit code 1 from systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem 2024-09-17T00:25:25.503 INFO:journalctl@ceph.mon.c.smithi002.stdout: systemctl: stderr Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:25:25.503 INFO:journalctl@ceph.mon.c.smithi002.stdout: systemctl: stderr See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:25:25.503 INFO:journalctl@ceph.mon.c.smithi002.stdout: Traceback (most recent call last): 2024-09-17T00:25:25.503 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-17T00:25:25.503 INFO:journalctl@ceph.mon.c.smithi002.stdout: return _run_code(code, main_globals, None, 2024-09-17T00:25:25.504 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-17T00:25:25.504 INFO:journalctl@ceph.mon.c.smithi002.stdout: exec(code, run_globals) 2024-09-17T00:25:25.504 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5579, in 2024-09-17T00:25:25.504 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5567, in main 2024-09-17T00:25:25.504 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3049, in command_deploy_from 2024-09-17T00:25:25.504 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3084, in _common_deploy 2024-09-17T00:25:25.505 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3104, in _deploy_daemon_container 2024-09-17T00:25:25.505 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 1124, in deploy_daemon 2024-09-17T00:25:25.505 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-17T00:25:25.505 INFO:journalctl@ceph.mon.c.smithi002.stdout: RuntimeError: Failed command: systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem: Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:25:25.505 INFO:journalctl@ceph.mon.c.smithi002.stdout: See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:25:25.505 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:25:25 smithi002 ceph-mon[66732]: from='mon.1 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-09-17T00:25:25.506 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:25:25 smithi002 ceph-mon[66732]: Upgrade: unsafe to stop osd(s) at this time (14 PGs are or would become offline) 2024-09-17T00:25:25.506 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:25:25 smithi002 ceph-mon[66732]: pgmap v68: 161 pgs: 161 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-09-17T00:25:26.979 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:25:26 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: ::ffff:172.21.15.99 - - [17/Sep/2024:00:25:26] "GET /metrics HTTP/1.1" 200 38396 "" "Prometheus/2.43.0" 2024-09-17T00:25:27.468 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:25:27 smithi099 ceph-mon[56712]: pgmap v69: 161 pgs: 161 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-17T00:25:27.469 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:25:27 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:25:27.469 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:25:27 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T00:25:27.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:25:27 smithi002 ceph-mon[66732]: pgmap v69: 161 pgs: 161 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-17T00:25:27.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:25:27 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:25:27.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:25:27 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T00:25:27.480 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:25:27 smithi002 ceph-mon[63804]: pgmap v69: 161 pgs: 161 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-17T00:25:27.480 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:25:27 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:25:27.480 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:25:27 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T00:25:28.967 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:25:28 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[52757]: ts=2024-09-17T00:25:28.659Z caller=alerting.go:366 level=warn component="rule manager" alert="unsupported value type" msg="Expanding alert template failed" err="error executing template __alert_CephOSDDown: template: __alert_CephOSDDown:1:358: executing \"__alert_CephOSDDown\" at : error calling query: found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.2\", device_class=\"ssd\", hostname=\"smithi002\", instance=\"172.21.15.99:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.2\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"3aad27a2-7489-11ef-bceb-c7b262605968\", cluster_addr=\"172.21.15.2\", device_class=\"ssd\", hostname=\"smithi002\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.2\"}];many-to-many matching not allowed: matching labels must be unique on one side" data="unsupported value type" 2024-09-17T00:25:28.967 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:25:28 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[52757]: ts=2024-09-17T00:25:28.661Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd name=CephOSDFlapping index=13 msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.2\", device_class=\"ssd\", hostname=\"smithi002\", instance=\"172.21.15.99:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.2\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"3aad27a2-7489-11ef-bceb-c7b262605968\", cluster_addr=\"172.21.15.2\", device_class=\"ssd\", hostname=\"smithi002\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.2\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-17T00:25:29.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:25:29 smithi002 ceph-mon[66732]: pgmap v70: 161 pgs: 161 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 514 B/s rd, 0 op/s 2024-09-17T00:25:29.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:25:29 smithi002 ceph-mon[63804]: pgmap v70: 161 pgs: 161 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 514 B/s rd, 0 op/s 2024-09-17T00:25:29.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:25:29 smithi099 ceph-mon[56712]: pgmap v70: 161 pgs: 161 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 514 B/s rd, 0 op/s 2024-09-17T00:25:31.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:25:31 smithi002 ceph-mon[66732]: pgmap v71: 161 pgs: 161 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 454 B/s rd, 0 op/s 2024-09-17T00:25:31.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:25:31 smithi002 ceph-mon[63804]: pgmap v71: 161 pgs: 161 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 454 B/s rd, 0 op/s 2024-09-17T00:25:31.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:25:31 smithi099 ceph-mon[56712]: pgmap v71: 161 pgs: 161 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 454 B/s rd, 0 op/s 2024-09-17T00:25:33.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:25:32 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:25:33.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:25:32 smithi099 ceph-mon[56712]: pgmap v72: 161 pgs: 161 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-09-17T00:25:33.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:25:32 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:25:33.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:25:32 smithi002 ceph-mon[66732]: pgmap v72: 161 pgs: 161 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-09-17T00:25:33.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:25:32 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:25:33.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:25:32 smithi002 ceph-mon[63804]: pgmap v72: 161 pgs: 161 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-09-17T00:25:33.967 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:25:33 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[52757]: ts=2024-09-17T00:25:33.506Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes name=CephNodeDiskspaceWarning index=4 msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi002\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi002\", job=\"node\", machine=\"x86_64\", nodename=\"smithi002\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"3aad27a2-7489-11ef-bceb-c7b262605968\", domainname=\"(none)\", instance=\"smithi002\", job=\"node\", machine=\"x86_64\", nodename=\"smithi002\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-17T00:25:35.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:25:35 smithi099 ceph-mon[56712]: pgmap v73: 161 pgs: 161 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-09-17T00:25:35.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:25:35 smithi002 ceph-mon[66732]: pgmap v73: 161 pgs: 161 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-09-17T00:25:35.480 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:25:35 smithi002 ceph-mon[63804]: pgmap v73: 161 pgs: 161 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-09-17T00:25:36.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:25:36 smithi099 ceph-mon[56712]: from='client.? 172.21.15.2:0/2937714272' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-17T00:25:36.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:25:36 smithi002 ceph-mon[66732]: from='client.? 172.21.15.2:0/2937714272' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-17T00:25:36.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:25:36 smithi002 ceph-mon[63804]: from='client.? 172.21.15.2:0/2937714272' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-17T00:25:36.979 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:25:36 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: ::ffff:172.21.15.99 - - [17/Sep/2024:00:25:36] "GET /metrics HTTP/1.1" 200 38399 "" "Prometheus/2.43.0" 2024-09-17T00:25:37.470 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:25:37 smithi099 ceph-mon[56712]: pgmap v74: 161 pgs: 161 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:25:37.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:25:37 smithi002 ceph-mon[63804]: pgmap v74: 161 pgs: 161 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:25:37.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:25:37 smithi002 ceph-mon[66732]: pgmap v74: 161 pgs: 161 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:25:38.967 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:25:38 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[52757]: ts=2024-09-17T00:25:38.660Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd name=CephOSDFlapping index=13 msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.1.1-330-gc2ddf972 (c2ddf9721ef254645201cdbabd9c8f49c30a586f) squid (stable)\", cluster=\"3aad27a2-7489-11ef-bceb-c7b262605968\", cluster_addr=\"172.21.15.2\", device_class=\"ssd\", hostname=\"smithi002\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.2\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.2\", device_class=\"ssd\", hostname=\"smithi002\", instance=\"172.21.15.99:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.2\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-17T00:25:39.355 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:25:39 smithi002 ceph-mon[66732]: pgmap v75: 161 pgs: 161 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-09-17T00:25:39.355 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:25:39 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-09-17T00:25:39.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:25:39 smithi002 ceph-mon[63804]: pgmap v75: 161 pgs: 161 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-09-17T00:25:39.356 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:25:39 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-09-17T00:25:39.466 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:25:39 smithi099 ceph-mon[56712]: pgmap v75: 161 pgs: 161 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-09-17T00:25:39.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:25:39 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-09-17T00:25:40.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:25:40 smithi099 ceph-mon[56712]: from='mon.1 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-09-17T00:25:40.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:25:40 smithi099 ceph-mon[56712]: Upgrade: osd.1 is safe to restart 2024-09-17T00:25:40.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:25:40 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:25:40.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:25:40 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-09-17T00:25:40.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:25:40 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:25:40.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:25:40 smithi002 ceph-mon[63804]: from='mon.1 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-09-17T00:25:40.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:25:40 smithi002 ceph-mon[63804]: Upgrade: osd.1 is safe to restart 2024-09-17T00:25:40.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:25:40 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:25:40.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:25:40 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-09-17T00:25:40.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:25:40 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:25:40.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:25:40 smithi002 ceph-mon[66732]: from='mon.1 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-09-17T00:25:40.480 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:25:40 smithi002 ceph-mon[66732]: Upgrade: osd.1 is safe to restart 2024-09-17T00:25:40.480 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:25:40 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:25:40.480 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:25:40 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-09-17T00:25:40.480 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:25:40 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:25:41.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:25:41 smithi099 ceph-mon[56712]: Upgrade: Updating osd.1 2024-09-17T00:25:41.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:25:41 smithi099 ceph-mon[56712]: Deploying daemon osd.1 on smithi002 2024-09-17T00:25:41.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:25:41 smithi099 ceph-mon[56712]: pgmap v76: 161 pgs: 161 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-09-17T00:25:41.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:25:41 smithi002 ceph-mon[63804]: Upgrade: Updating osd.1 2024-09-17T00:25:41.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:25:41 smithi002 ceph-mon[63804]: Deploying daemon osd.1 on smithi002 2024-09-17T00:25:41.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:25:41 smithi002 ceph-mon[63804]: pgmap v76: 161 pgs: 161 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-09-17T00:25:41.480 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:25:41 smithi002 ceph-mon[66732]: Upgrade: Updating osd.1 2024-09-17T00:25:41.480 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:25:41 smithi002 ceph-mon[66732]: Deploying daemon osd.1 on smithi002 2024-09-17T00:25:41.480 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:25:41 smithi002 ceph-mon[66732]: pgmap v76: 161 pgs: 161 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-09-17T00:25:42.421 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:25:42 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T00:25:42.422 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:25:42 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T00:25:42.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:25:42 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T00:25:43.386 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:25:43 smithi002 ceph-mon[63804]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-09-17T00:25:43.386 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:25:43 smithi002 ceph-mon[66732]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-09-17T00:25:43.387 INFO:journalctl@ceph.osd.1.smithi002.stdout:Sep 17 00:25:43 smithi002 systemd[1]: Stopping Ceph osd.1 for 3aad27a2-7489-11ef-bceb-c7b262605968... 2024-09-17T00:25:43.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:25:43 smithi099 ceph-mon[56712]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-09-17T00:25:43.729 INFO:journalctl@ceph.osd.1.smithi002.stdout:Sep 17 00:25:43 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-1[33959]: 2024-09-17T00:25:43.386+0000 7f90f45b7700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-17T00:25:43.729 INFO:journalctl@ceph.osd.1.smithi002.stdout:Sep 17 00:25:43 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-1[33959]: 2024-09-17T00:25:43.387+0000 7f90f45b7700 -1 osd.1 117 *** Got signal Terminated *** 2024-09-17T00:25:43.729 INFO:journalctl@ceph.osd.1.smithi002.stdout:Sep 17 00:25:43 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-1[33959]: 2024-09-17T00:25:43.387+0000 7f90f45b7700 -1 osd.1 117 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-17T00:25:43.967 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:25:43 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[52757]: ts=2024-09-17T00:25:43.506Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes name=CephNodeDiskspaceWarning index=4 msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi002\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi002\", job=\"node\", machine=\"x86_64\", nodename=\"smithi002\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"3aad27a2-7489-11ef-bceb-c7b262605968\", domainname=\"(none)\", instance=\"smithi002\", job=\"node\", machine=\"x86_64\", nodename=\"smithi002\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-17T00:25:44.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:25:44 smithi099 ceph-mon[56712]: osd.1 marked itself down and dead 2024-09-17T00:25:44.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:25:44 smithi002 ceph-mon[63804]: osd.1 marked itself down and dead 2024-09-17T00:25:44.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:25:44 smithi002 ceph-mon[66732]: osd.1 marked itself down and dead 2024-09-17T00:25:44.902 INFO:journalctl@ceph.osd.1.smithi002.stdout:Sep 17 00:25:44 smithi002 podman[73684]: 2024-09-17 00:25:44.595852104 +0000 UTC m=+1.340054105 container died 7cb62c311c7ddc2c8000c785d02112486a754b51e0aedaac1362d53a10faf740 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-1, vcs-type=git, CEPH_POINT_RELEASE=-17.2.0, io.openshift.tags=base centos centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., RELEASE=HEAD, vendor=Red Hat, Inc., release=754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.component=centos-stream-container, version=8, name=centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, distribution-scope=public, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_CLEAN=True, io.k8s.display-name=CentOS Stream 8, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, maintainer=Guillaume Abrioux , GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.buildah.version=1.19.8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, build-date=2022-05-03T08:36:31.336870, io.openshift.expose-services=, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, ceph=True) 2024-09-17T00:25:45.157 INFO:journalctl@ceph.osd.1.smithi002.stdout:Sep 17 00:25:45 smithi002 podman[73684]: 2024-09-17 00:25:45.004426156 +0000 UTC m=+1.748628156 container cleanup 7cb62c311c7ddc2c8000c785d02112486a754b51e0aedaac1362d53a10faf740 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-1, io.k8s.display-name=CentOS Stream 8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_CLEAN=True, io.openshift.expose-services=, vcs-type=git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_BRANCH=HEAD, release=754, RELEASE=HEAD, com.redhat.component=centos-stream-container, architecture=x86_64, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=-17.2.0, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, name=centos-stream, io.openshift.tags=base centos centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vendor=Red Hat, Inc., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.19.8, version=8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, build-date=2022-05-03T08:36:31.336870, distribution-scope=public, ceph=True) 2024-09-17T00:25:45.157 INFO:journalctl@ceph.osd.1.smithi002.stdout:Sep 17 00:25:45 smithi002 bash[73684]: ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-1 2024-09-17T00:25:45.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:25:45 smithi099 ceph-mon[56712]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-09-17T00:25:45.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:25:45 smithi099 ceph-mon[56712]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-17T00:25:45.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:25:45 smithi099 ceph-mon[56712]: osdmap e118: 8 total, 7 up, 8 in 2024-09-17T00:25:45.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:25:45 smithi002 ceph-mon[66732]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-09-17T00:25:45.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:25:45 smithi002 ceph-mon[66732]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-17T00:25:45.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:25:45 smithi002 ceph-mon[66732]: osdmap e118: 8 total, 7 up, 8 in 2024-09-17T00:25:45.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:25:45 smithi002 ceph-mon[63804]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-09-17T00:25:45.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:25:45 smithi002 ceph-mon[63804]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-17T00:25:45.480 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:25:45 smithi002 ceph-mon[63804]: osdmap e118: 8 total, 7 up, 8 in 2024-09-17T00:25:45.480 INFO:journalctl@ceph.osd.1.smithi002.stdout:Sep 17 00:25:45 smithi002 podman[73702]: 2024-09-17 00:25:45.247330817 +0000 UTC m=+0.650865849 container remove 7cb62c311c7ddc2c8000c785d02112486a754b51e0aedaac1362d53a10faf740 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-1, build-date=2022-05-03T08:36:31.336870, release=754, GIT_BRANCH=HEAD, io.openshift.expose-services=, RELEASE=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, name=centos-stream, version=8, com.redhat.component=centos-stream-container, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vcs-type=git, vendor=Red Hat, Inc., io.buildah.version=1.19.8, io.k8s.display-name=CentOS Stream 8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=-17.2.0, ceph=True, architecture=x86_64, distribution-scope=public, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, io.openshift.tags=base centos centos-stream) 2024-09-17T00:25:45.942 INFO:journalctl@ceph.osd.1.smithi002.stdout:Sep 17 00:25:45 smithi002 podman[73789]: 2024-09-17 00:25:45.590326067 +0000 UTC m=+0.016672380 image pull ebb0615149def4a3179c82c0cc47af75808300abb9f1142788e5e605a7ac0e5b quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f 2024-09-17T00:25:45.943 INFO:journalctl@ceph.osd.1.smithi002.stdout:Sep 17 00:25:45 smithi002 podman[73789]: 2024-09-17 00:25:45.698028538 +0000 UTC m=+0.124374826 container create 92f7116f61c5c4373e48491a23e56b12ccf0fb9c6be3d4c7e98b5da7d0ae17f8 (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-1-deactivate, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-c2ddf97, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-09-17T00:25:46.225 INFO:journalctl@ceph.osd.1.smithi002.stdout:Sep 17 00:25:45 smithi002 podman[73789]: 2024-09-17 00:25:45.968790433 +0000 UTC m=+0.395136730 container init 92f7116f61c5c4373e48491a23e56b12ccf0fb9c6be3d4c7e98b5da7d0ae17f8 (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-1-deactivate, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, RELEASE=squid-c2ddf97, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD) 2024-09-17T00:25:46.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Sep 17 00:25:45 smithi002 podman[73789]: 2024-09-17 00:25:45.972738141 +0000 UTC m=+0.399084427 container start 92f7116f61c5c4373e48491a23e56b12ccf0fb9c6be3d4c7e98b5da7d0ae17f8 (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-1-deactivate, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=squid-c2ddf97, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2) 2024-09-17T00:25:46.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Sep 17 00:25:46 smithi002 podman[73789]: 2024-09-17 00:25:46.050575935 +0000 UTC m=+0.476922228 container attach 92f7116f61c5c4373e48491a23e56b12ccf0fb9c6be3d4c7e98b5da7d0ae17f8 (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-1-deactivate, io.buildah.version=1.37.2, RELEASE=squid-c2ddf97, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=) 2024-09-17T00:25:46.226 INFO:journalctl@ceph.osd.1.smithi002.stdout:Sep 17 00:25:46 smithi002 podman[73789]: 2024-09-17 00:25:46.171026228 +0000 UTC m=+0.597372529 container died 92f7116f61c5c4373e48491a23e56b12ccf0fb9c6be3d4c7e98b5da7d0ae17f8 (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, RELEASE=squid-c2ddf97, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-09-17T00:25:46.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:25:46 smithi002 ceph-mon[63804]: from='client.44332 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:25:46.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:25:46 smithi002 ceph-mon[63804]: osdmap e119: 8 total, 7 up, 8 in 2024-09-17T00:25:46.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:25:46 smithi002 ceph-mon[66732]: from='client.44332 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:25:46.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:25:46 smithi002 ceph-mon[66732]: osdmap e119: 8 total, 7 up, 8 in 2024-09-17T00:25:46.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:25:46 smithi099 ceph-mon[56712]: from='client.44332 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:25:46.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:25:46 smithi099 ceph-mon[56712]: osdmap e119: 8 total, 7 up, 8 in 2024-09-17T00:25:46.979 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:25:46 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: ::ffff:172.21.15.99 - - [17/Sep/2024:00:25:46] "GET /metrics HTTP/1.1" 200 38325 "" "Prometheus/2.43.0" 2024-09-17T00:25:46.980 INFO:journalctl@ceph.osd.1.smithi002.stdout:Sep 17 00:25:46 smithi002 podman[73811]: 2024-09-17 00:25:46.779709724 +0000 UTC m=+0.602756020 container remove 92f7116f61c5c4373e48491a23e56b12ccf0fb9c6be3d4c7e98b5da7d0ae17f8 (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-1-deactivate, ceph=True, io.buildah.version=1.37.2, RELEASE=squid-c2ddf97, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-09-17T00:25:46.980 INFO:journalctl@ceph.osd.1.smithi002.stdout:Sep 17 00:25:46 smithi002 systemd[1]: ceph-3aad27a2-7489-11ef-bceb-c7b262605968@osd.1.service: Deactivated successfully. 2024-09-17T00:25:46.980 INFO:journalctl@ceph.osd.1.smithi002.stdout:Sep 17 00:25:46 smithi002 systemd[1]: Stopped Ceph osd.1 for 3aad27a2-7489-11ef-bceb-c7b262605968. 2024-09-17T00:25:46.980 INFO:journalctl@ceph.osd.1.smithi002.stdout:Sep 17 00:25:46 smithi002 systemd[1]: ceph-3aad27a2-7489-11ef-bceb-c7b262605968@osd.1.service: Consumed 8.976s CPU time. 2024-09-17T00:25:47.362 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:25:47 smithi002 ceph-mon[66732]: pgmap v81: 161 pgs: 20 stale+active+clean, 141 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-17T00:25:47.363 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:25:47 smithi002 ceph-mon[63804]: pgmap v81: 161 pgs: 20 stale+active+clean, 141 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-17T00:25:47.363 INFO:journalctl@ceph.osd.1.smithi002.stdout:Sep 17 00:25:47 smithi002 systemd[1]: Starting Ceph osd.1 for 3aad27a2-7489-11ef-bceb-c7b262605968... 2024-09-17T00:25:47.363 INFO:journalctl@ceph.osd.1.smithi002.stdout:Sep 17 00:25:47 smithi002 podman[73908]: 2024-09-17 00:25:47.127188788 +0000 UTC m=+0.016519211 image pull ebb0615149def4a3179c82c0cc47af75808300abb9f1142788e5e605a7ac0e5b quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f 2024-09-17T00:25:47.364 INFO:journalctl@ceph.osd.1.smithi002.stdout:Sep 17 00:25:47 smithi002 podman[73908]: 2024-09-17 00:25:47.255628655 +0000 UTC m=+0.144959068 container create 4615cd544a560a6d467277aee093d76950f840454f93ac0556d1f4f54b6dd70c (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-1-activate, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-c2ddf97, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD) 2024-09-17T00:25:47.626 INFO:journalctl@ceph.osd.1.smithi002.stdout:Sep 17 00:25:47 smithi002 podman[73908]: 2024-09-17 00:25:47.491464532 +0000 UTC m=+0.380794953 container init 4615cd544a560a6d467277aee093d76950f840454f93ac0556d1f4f54b6dd70c (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-1-activate, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-c2ddf97, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-09-17T00:25:47.626 INFO:journalctl@ceph.osd.1.smithi002.stdout:Sep 17 00:25:47 smithi002 podman[73908]: 2024-09-17 00:25:47.495009069 +0000 UTC m=+0.384339482 container start 4615cd544a560a6d467277aee093d76950f840454f93ac0556d1f4f54b6dd70c (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-1-activate, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, RELEASE=squid-c2ddf97, maintainer=Guillaume Abrioux ) 2024-09-17T00:25:47.626 INFO:journalctl@ceph.osd.1.smithi002.stdout:Sep 17 00:25:47 smithi002 podman[73908]: 2024-09-17 00:25:47.591436867 +0000 UTC m=+0.480767295 container attach 4615cd544a560a6d467277aee093d76950f840454f93ac0556d1f4f54b6dd70c (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-1-activate, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=squid-c2ddf97, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True) 2024-09-17T00:25:47.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:25:47 smithi099 ceph-mon[56712]: pgmap v81: 161 pgs: 20 stale+active+clean, 141 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-17T00:25:47.979 INFO:journalctl@ceph.osd.1.smithi002.stdout:Sep 17 00:25:47 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-1-activate[73923]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-17T00:25:47.979 INFO:journalctl@ceph.osd.1.smithi002.stdout:Sep 17 00:25:47 smithi002 bash[73908]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-17T00:25:47.980 INFO:journalctl@ceph.osd.1.smithi002.stdout:Sep 17 00:25:47 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-1-activate[73923]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-17T00:25:47.980 INFO:journalctl@ceph.osd.1.smithi002.stdout:Sep 17 00:25:47 smithi002 bash[73908]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-17T00:25:48.656 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:25:48 smithi099 ceph-mon[56712]: Health check failed: Degraded data redundancy: 104/726 objects degraded (14.325%), 23 pgs degraded (PG_DEGRADED) 2024-09-17T00:25:48.728 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:25:48 smithi002 ceph-mon[66732]: Health check failed: Degraded data redundancy: 104/726 objects degraded (14.325%), 23 pgs degraded (PG_DEGRADED) 2024-09-17T00:25:48.728 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:25:48 smithi002 ceph-mon[63804]: Health check failed: Degraded data redundancy: 104/726 objects degraded (14.325%), 23 pgs degraded (PG_DEGRADED) 2024-09-17T00:25:48.967 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:25:48 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[52757]: ts=2024-09-17T00:25:48.659Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd name=CephOSDFlapping index=13 msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.1.1-330-gc2ddf972 (c2ddf9721ef254645201cdbabd9c8f49c30a586f) squid (stable)\", cluster=\"3aad27a2-7489-11ef-bceb-c7b262605968\", cluster_addr=\"172.21.15.2\", device_class=\"ssd\", hostname=\"smithi002\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.2\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.2\", device_class=\"ssd\", hostname=\"smithi002\", instance=\"172.21.15.99:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.2\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-17T00:25:48.979 INFO:journalctl@ceph.osd.1.smithi002.stdout:Sep 17 00:25:48 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-1-activate[73923]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-17T00:25:48.979 INFO:journalctl@ceph.osd.1.smithi002.stdout:Sep 17 00:25:48 smithi002 bash[73908]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-17T00:25:48.979 INFO:journalctl@ceph.osd.1.smithi002.stdout:Sep 17 00:25:48 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-1-activate[73923]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-09-17T00:25:48.979 INFO:journalctl@ceph.osd.1.smithi002.stdout:Sep 17 00:25:48 smithi002 bash[73908]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-09-17T00:25:49.279 INFO:journalctl@ceph.osd.1.smithi002.stdout:Sep 17 00:25:49 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-1-activate[73923]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-09-17T00:25:49.280 INFO:journalctl@ceph.osd.1.smithi002.stdout:Sep 17 00:25:49 smithi002 bash[73908]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-09-17T00:25:49.280 INFO:journalctl@ceph.osd.1.smithi002.stdout:Sep 17 00:25:49 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-1-activate[73923]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-09-17T00:25:49.280 INFO:journalctl@ceph.osd.1.smithi002.stdout:Sep 17 00:25:49 smithi002 bash[73908]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-09-17T00:25:49.280 INFO:journalctl@ceph.osd.1.smithi002.stdout:Sep 17 00:25:49 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-1-activate[73923]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-09-17T00:25:49.280 INFO:journalctl@ceph.osd.1.smithi002.stdout:Sep 17 00:25:49 smithi002 bash[73908]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-09-17T00:25:49.280 INFO:journalctl@ceph.osd.1.smithi002.stdout:Sep 17 00:25:49 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-1-activate[73923]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-17T00:25:49.280 INFO:journalctl@ceph.osd.1.smithi002.stdout:Sep 17 00:25:49 smithi002 bash[73908]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-17T00:25:49.280 INFO:journalctl@ceph.osd.1.smithi002.stdout:Sep 17 00:25:49 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-1-activate[73923]: --> ceph-volume raw activate successful for osd ID: 1 2024-09-17T00:25:49.280 INFO:journalctl@ceph.osd.1.smithi002.stdout:Sep 17 00:25:49 smithi002 bash[73908]: --> ceph-volume raw activate successful for osd ID: 1 2024-09-17T00:25:49.280 INFO:journalctl@ceph.osd.1.smithi002.stdout:Sep 17 00:25:49 smithi002 podman[73908]: 2024-09-17 00:25:49.061368162 +0000 UTC m=+1.950698584 container died 4615cd544a560a6d467277aee093d76950f840454f93ac0556d1f4f54b6dd70c (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-1-activate, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, RELEASE=squid-c2ddf97, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2) 2024-09-17T00:25:49.659 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:25:49 smithi002 ceph-mon[63804]: pgmap v82: 161 pgs: 40 active+undersized, 23 active+undersized+degraded, 98 active+clean; 585 KiB data, 121 MiB used, 715 GiB / 715 GiB avail; 104/726 objects degraded (14.325%) 2024-09-17T00:25:49.659 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:25:49 smithi002 ceph-mon[66732]: pgmap v82: 161 pgs: 40 active+undersized, 23 active+undersized+degraded, 98 active+clean; 585 KiB data, 121 MiB used, 715 GiB / 715 GiB avail; 104/726 objects degraded (14.325%) 2024-09-17T00:25:49.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:25:49 smithi099 ceph-mon[56712]: pgmap v82: 161 pgs: 40 active+undersized, 23 active+undersized+degraded, 98 active+clean; 585 KiB data, 121 MiB used, 715 GiB / 715 GiB avail; 104/726 objects degraded (14.325%) 2024-09-17T00:25:49.979 INFO:journalctl@ceph.osd.1.smithi002.stdout:Sep 17 00:25:49 smithi002 podman[74066]: 2024-09-17 00:25:49.662047907 +0000 UTC m=+0.593434691 container remove 4615cd544a560a6d467277aee093d76950f840454f93ac0556d1f4f54b6dd70c (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-1-activate, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-c2ddf97, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-17T00:25:49.979 INFO:journalctl@ceph.osd.1.smithi002.stdout:Sep 17 00:25:49 smithi002 podman[74126]: 2024-09-17 00:25:49.786732036 +0000 UTC m=+0.017563407 image pull ebb0615149def4a3179c82c0cc47af75808300abb9f1142788e5e605a7ac0e5b quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f 2024-09-17T00:25:49.979 INFO:journalctl@ceph.osd.1.smithi002.stdout:Sep 17 00:25:49 smithi002 podman[74126]: 2024-09-17 00:25:49.887862547 +0000 UTC m=+0.118693908 container create 3dbc23d57c5342b75bf62f260234f2a9b2d29ffb9fd01765e9bb0b5589299b23 (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-1, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, ceph=True, RELEASE=squid-c2ddf97, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-17T00:25:50.271 INFO:journalctl@ceph.osd.1.smithi002.stdout:Sep 17 00:25:50 smithi002 podman[74126]: 2024-09-17 00:25:50.248628741 +0000 UTC m=+0.479460117 container init 3dbc23d57c5342b75bf62f260234f2a9b2d29ffb9fd01765e9bb0b5589299b23 (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-1, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=squid-c2ddf97, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=) 2024-09-17T00:25:50.272 INFO:journalctl@ceph.osd.1.smithi002.stdout:Sep 17 00:25:50 smithi002 podman[74126]: 2024-09-17 00:25:50.252553875 +0000 UTC m=+0.483385236 container start 3dbc23d57c5342b75bf62f260234f2a9b2d29ffb9fd01765e9bb0b5589299b23 (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-1, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=squid-c2ddf97, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, ceph=True) 2024-09-17T00:25:50.729 INFO:journalctl@ceph.osd.1.smithi002.stdout:Sep 17 00:25:50 smithi002 ceph-osd[74145]: -- 172.21.15.2:0/2752808616 <== mon.1 v2:172.21.15.2:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55d7b0027a40 con 0x55d7b0dee400 2024-09-17T00:25:50.729 INFO:journalctl@ceph.osd.1.smithi002.stdout:Sep 17 00:25:50 smithi002 bash[74126]: 3dbc23d57c5342b75bf62f260234f2a9b2d29ffb9fd01765e9bb0b5589299b23 2024-09-17T00:25:50.729 INFO:journalctl@ceph.osd.1.smithi002.stdout:Sep 17 00:25:50 smithi002 systemd[1]: Started Ceph osd.1 for 3aad27a2-7489-11ef-bceb-c7b262605968. 2024-09-17T00:25:51.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:25:51 smithi099 ceph-mon[56712]: pgmap v83: 161 pgs: 40 active+undersized, 23 active+undersized+degraded, 98 active+clean; 585 KiB data, 121 MiB used, 715 GiB / 715 GiB avail; 104/726 objects degraded (14.325%) 2024-09-17T00:25:51.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:25:51 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:25:51.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:25:51 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:25:51.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:25:51 smithi002 ceph-mon[66732]: pgmap v83: 161 pgs: 40 active+undersized, 23 active+undersized+degraded, 98 active+clean; 585 KiB data, 121 MiB used, 715 GiB / 715 GiB avail; 104/726 objects degraded (14.325%) 2024-09-17T00:25:51.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:25:51 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:25:51.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:25:51 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:25:51.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:25:51 smithi002 ceph-mon[63804]: pgmap v83: 161 pgs: 40 active+undersized, 23 active+undersized+degraded, 98 active+clean; 585 KiB data, 121 MiB used, 715 GiB / 715 GiB avail; 104/726 objects degraded (14.325%) 2024-09-17T00:25:51.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:25:51 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:25:51.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:25:51 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:25:51.973 INFO:teuthology.orchestra.run.smithi002.stdout:true 2024-09-17T00:25:52.229 INFO:journalctl@ceph.osd.1.smithi002.stdout:Sep 17 00:25:51 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-1[74140]: 2024-09-17T00:25:51.885+0000 7fd008b56640 -1 Falling back to public interface 2024-09-17T00:25:52.698 INFO:teuthology.orchestra.run.smithi002.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-17T00:25:52.698 INFO:teuthology.orchestra.run.smithi002.stdout:alertmanager.a smithi002 *:9093,9094 running (3m) 36s ago 8m 26.3M - 0.25.0 c8568f914cd2 a21e13b3d677 2024-09-17T00:25:52.698 INFO:teuthology.orchestra.run.smithi002.stdout:grafana.a smithi099 *:3000 running (3m) 85s ago 8m 46.9M - 8.3.5 dad864ee21e9 bafb824a3c08 2024-09-17T00:25:52.698 INFO:teuthology.orchestra.run.smithi002.stdout:iscsi.foo.smithi002.vmouem smithi002 running (54s) 36s ago 7m 50.3M - 3.5 e1d6a67b021e 9d37e0963524 2024-09-17T00:25:52.698 INFO:teuthology.orchestra.run.smithi002.stdout:mgr.x smithi099 *:8443,9283,8765 running (2m) 85s ago 13m 439M - 19.1.1-330-gc2ddf972 ebb0615149de 6aa2ccef9ca0 2024-09-17T00:25:52.698 INFO:teuthology.orchestra.run.smithi002.stdout:mgr.y smithi002 *:8443,9283,8765 running (3m) 36s ago 14m 500M - 19.1.1-330-gc2ddf972 ebb0615149de 8838897e8449 2024-09-17T00:25:52.698 INFO:teuthology.orchestra.run.smithi002.stdout:mon.a smithi002 running (2m) 36s ago 15m 47.3M 2048M 19.1.1-330-gc2ddf972 ebb0615149de a035def53ef6 2024-09-17T00:25:52.698 INFO:teuthology.orchestra.run.smithi002.stdout:mon.b smithi099 running (96s) 85s ago 13m 17.5M 2048M 19.1.1-330-gc2ddf972 ebb0615149de aab3ab73a850 2024-09-17T00:25:52.698 INFO:teuthology.orchestra.run.smithi002.stdout:mon.c smithi002 running (113s) 36s ago 13m 38.9M 2048M 19.1.1-330-gc2ddf972 ebb0615149de 8a510255ae66 2024-09-17T00:25:52.699 INFO:teuthology.orchestra.run.smithi002.stdout:node-exporter.a smithi002 *:9100 running (3m) 36s ago 9m 14.7M - 1.5.0 0da6a335fe13 1030ae01bb32 2024-09-17T00:25:52.699 INFO:teuthology.orchestra.run.smithi002.stdout:node-exporter.b smithi099 *:9100 running (3m) 85s ago 9m 16.8M - 1.5.0 0da6a335fe13 01fcd35dca8a 2024-09-17T00:25:52.699 INFO:teuthology.orchestra.run.smithi002.stdout:osd.0 smithi002 running (42s) 36s ago 12m 12.1M 2737M 19.1.1-330-gc2ddf972 ebb0615149de 8345719cb6f0 2024-09-17T00:25:52.699 INFO:teuthology.orchestra.run.smithi002.stdout:osd.1 smithi002 starting - - - 2737M 2024-09-17T00:25:52.699 INFO:teuthology.orchestra.run.smithi002.stdout:osd.2 smithi002 running (12m) 36s ago 12m 53.3M 2737M 17.2.0 e1d6a67b021e eafddacb955e 2024-09-17T00:25:52.699 INFO:teuthology.orchestra.run.smithi002.stdout:osd.3 smithi002 running (11m) 36s ago 11m 57.0M 2737M 17.2.0 e1d6a67b021e fd99d3764412 2024-09-17T00:25:52.699 INFO:teuthology.orchestra.run.smithi002.stdout:osd.4 smithi099 running (11m) 85s ago 11m 55.4M 2992M 17.2.0 e1d6a67b021e 0af1827b658d 2024-09-17T00:25:52.699 INFO:teuthology.orchestra.run.smithi002.stdout:osd.5 smithi099 running (10m) 85s ago 10m 54.2M 2992M 17.2.0 e1d6a67b021e 4f7ef3aefc58 2024-09-17T00:25:52.699 INFO:teuthology.orchestra.run.smithi002.stdout:osd.6 smithi099 running (10m) 85s ago 10m 51.0M 2992M 17.2.0 e1d6a67b021e 3213f5e6952a 2024-09-17T00:25:52.699 INFO:teuthology.orchestra.run.smithi002.stdout:osd.7 smithi099 running (10m) 85s ago 10m 56.6M 2992M 17.2.0 e1d6a67b021e 63be3a1ed0a1 2024-09-17T00:25:52.699 INFO:teuthology.orchestra.run.smithi002.stdout:prometheus.a smithi099 *:9095 running (2m) 85s ago 9m 50.1M - 2.43.0 a07b618ecd1d bf32c2ae11dc 2024-09-17T00:25:52.699 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.foo.smithi002.uwdhcs smithi002 *:8000 running (7m) 36s ago 7m 75.6M - 17.2.0 e1d6a67b021e 9c35813f459a 2024-09-17T00:25:52.699 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.foo.smithi099.utzfap smithi099 *:8000 running (7m) 85s ago 7m 76.0M - 17.2.0 e1d6a67b021e 34ec989160a6 2024-09-17T00:25:52.699 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.smpl.smithi002.rqnxii smithi002 *:80 running (7m) 36s ago 7m 74.0M - 17.2.0 e1d6a67b021e 0e180d0afb99 2024-09-17T00:25:52.700 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.smpl.smithi099.nzvkpm smithi099 *:80 running (7m) 85s ago 7m 76.5M - 17.2.0 e1d6a67b021e 6a068e82ac72 2024-09-17T00:25:53.119 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-09-17T00:25:53.122 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": { 2024-09-17T00:25:53.122 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 19.1.1-330-gc2ddf972 (c2ddf9721ef254645201cdbabd9c8f49c30a586f) squid (stable)": 3 2024-09-17T00:25:53.122 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-09-17T00:25:53.122 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": { 2024-09-17T00:25:53.122 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 19.1.1-330-gc2ddf972 (c2ddf9721ef254645201cdbabd9c8f49c30a586f) squid (stable)": 2 2024-09-17T00:25:53.122 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-09-17T00:25:53.122 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": { 2024-09-17T00:25:53.122 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 6, 2024-09-17T00:25:53.122 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 19.1.1-330-gc2ddf972 (c2ddf9721ef254645201cdbabd9c8f49c30a586f) squid (stable)": 1 2024-09-17T00:25:53.122 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-09-17T00:25:53.122 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw": { 2024-09-17T00:25:53.122 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-09-17T00:25:53.123 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-09-17T00:25:53.123 INFO:teuthology.orchestra.run.smithi002.stdout: "overall": { 2024-09-17T00:25:53.123 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 10, 2024-09-17T00:25:53.123 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 19.1.1-330-gc2ddf972 (c2ddf9721ef254645201cdbabd9c8f49c30a586f) squid (stable)": 6 2024-09-17T00:25:53.123 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-09-17T00:25:53.123 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-09-17T00:25:53.482 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-09-17T00:25:53.482 INFO:teuthology.orchestra.run.smithi002.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f", 2024-09-17T00:25:53.482 INFO:teuthology.orchestra.run.smithi002.stdout: "in_progress": true, 2024-09-17T00:25:53.482 INFO:teuthology.orchestra.run.smithi002.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-17T00:25:53.483 INFO:teuthology.orchestra.run.smithi002.stdout: "services_complete": [ 2024-09-17T00:25:53.483 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr", 2024-09-17T00:25:53.483 INFO:teuthology.orchestra.run.smithi002.stdout: "mon" 2024-09-17T00:25:53.483 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-09-17T00:25:53.483 INFO:teuthology.orchestra.run.smithi002.stdout: "progress": "6/23 daemons upgraded", 2024-09-17T00:25:53.483 INFO:teuthology.orchestra.run.smithi002.stdout: "message": "Currently upgrading osd daemons", 2024-09-17T00:25:53.483 INFO:teuthology.orchestra.run.smithi002.stdout: "is_paused": false 2024-09-17T00:25:53.483 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-09-17T00:25:53.637 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:25:53 smithi002 ceph-mon[66732]: from='client.34443 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:25:53.637 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:25:53 smithi002 ceph-mon[66732]: pgmap v84: 161 pgs: 40 active+undersized, 23 active+undersized+degraded, 98 active+clean; 585 KiB data, 121 MiB used, 715 GiB / 715 GiB avail; 104/726 objects degraded (14.325%) 2024-09-17T00:25:53.637 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:25:53 smithi002 ceph-mon[66732]: from='client.44356 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:25:53.637 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:25:53 smithi002 ceph-mon[66732]: from='client.? 172.21.15.2:0/2581258129' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:25:53.637 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:25:53 smithi002 ceph-mon[63804]: from='client.34443 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:25:53.637 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:25:53 smithi002 ceph-mon[63804]: pgmap v84: 161 pgs: 40 active+undersized, 23 active+undersized+degraded, 98 active+clean; 585 KiB data, 121 MiB used, 715 GiB / 715 GiB avail; 104/726 objects degraded (14.325%) 2024-09-17T00:25:53.637 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:25:53 smithi002 ceph-mon[63804]: from='client.44356 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:25:53.637 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:25:53 smithi002 ceph-mon[63804]: from='client.? 172.21.15.2:0/2581258129' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:25:53.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:25:53 smithi099 ceph-mon[56712]: from='client.34443 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:25:53.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:25:53 smithi099 ceph-mon[56712]: pgmap v84: 161 pgs: 40 active+undersized, 23 active+undersized+degraded, 98 active+clean; 585 KiB data, 121 MiB used, 715 GiB / 715 GiB avail; 104/726 objects degraded (14.325%) 2024-09-17T00:25:53.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:25:53 smithi099 ceph-mon[56712]: from='client.44356 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:25:53.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:25:53 smithi099 ceph-mon[56712]: from='client.? 172.21.15.2:0/2581258129' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:25:53.717 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:25:53 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[52757]: ts=2024-09-17T00:25:53.506Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes name=CephNodeDiskspaceWarning index=4 msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi002\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi002\", job=\"node\", machine=\"x86_64\", nodename=\"smithi002\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"3aad27a2-7489-11ef-bceb-c7b262605968\", domainname=\"(none)\", instance=\"smithi002\", job=\"node\", machine=\"x86_64\", nodename=\"smithi002\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-17T00:25:53.906 INFO:teuthology.orchestra.run.smithi002.stdout:HEALTH_WARN 1 osds down; Degraded data redundancy: 104/726 objects degraded (14.325%), 23 pgs degraded 2024-09-17T00:25:53.906 INFO:teuthology.orchestra.run.smithi002.stdout:[WRN] OSD_DOWN: 1 osds down 2024-09-17T00:25:53.906 INFO:teuthology.orchestra.run.smithi002.stdout: osd.1 (root=default,host=smithi002) is down 2024-09-17T00:25:53.906 INFO:teuthology.orchestra.run.smithi002.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 104/726 objects degraded (14.325%), 23 pgs degraded 2024-09-17T00:25:53.906 INFO:teuthology.orchestra.run.smithi002.stdout: pg 2.2 is active+undersized+degraded, acting [5,6] 2024-09-17T00:25:53.907 INFO:teuthology.orchestra.run.smithi002.stdout: pg 2.9 is active+undersized+degraded, acting [7,3] 2024-09-17T00:25:53.907 INFO:teuthology.orchestra.run.smithi002.stdout: pg 2.a is active+undersized+degraded, acting [3,7] 2024-09-17T00:25:53.907 INFO:teuthology.orchestra.run.smithi002.stdout: pg 2.11 is active+undersized+degraded, acting [6,4] 2024-09-17T00:25:53.907 INFO:teuthology.orchestra.run.smithi002.stdout: pg 2.15 is active+undersized+degraded, acting [5,0] 2024-09-17T00:25:53.907 INFO:teuthology.orchestra.run.smithi002.stdout: pg 3.0 is active+undersized+degraded, acting [2,6] 2024-09-17T00:25:53.907 INFO:teuthology.orchestra.run.smithi002.stdout: pg 3.4 is active+undersized+degraded, acting [2,5] 2024-09-17T00:25:53.907 INFO:teuthology.orchestra.run.smithi002.stdout: pg 3.6 is active+undersized+degraded, acting [0,4] 2024-09-17T00:25:53.907 INFO:teuthology.orchestra.run.smithi002.stdout: pg 3.8 is active+undersized+degraded, acting [3,7] 2024-09-17T00:25:53.907 INFO:teuthology.orchestra.run.smithi002.stdout: pg 3.a is active+undersized+degraded, acting [6,4] 2024-09-17T00:25:53.907 INFO:teuthology.orchestra.run.smithi002.stdout: pg 3.e is active+undersized+degraded, acting [7,4] 2024-09-17T00:25:53.907 INFO:teuthology.orchestra.run.smithi002.stdout: pg 3.16 is active+undersized+degraded, acting [5,7] 2024-09-17T00:25:53.907 INFO:teuthology.orchestra.run.smithi002.stdout: pg 3.18 is active+undersized+degraded, acting [3,0] 2024-09-17T00:25:53.907 INFO:teuthology.orchestra.run.smithi002.stdout: pg 3.19 is active+undersized+degraded, acting [3,4] 2024-09-17T00:25:53.907 INFO:teuthology.orchestra.run.smithi002.stdout: pg 3.1a is active+undersized+degraded, acting [4,2] 2024-09-17T00:25:53.908 INFO:teuthology.orchestra.run.smithi002.stdout: pg 3.1c is active+undersized+degraded, acting [5,4] 2024-09-17T00:25:53.908 INFO:teuthology.orchestra.run.smithi002.stdout: pg 4.9 is active+undersized+degraded, acting [4,3] 2024-09-17T00:25:53.908 INFO:teuthology.orchestra.run.smithi002.stdout: pg 4.d is active+undersized+degraded, acting [4,2] 2024-09-17T00:25:53.908 INFO:teuthology.orchestra.run.smithi002.stdout: pg 4.e is active+undersized+degraded, acting [4,6] 2024-09-17T00:25:53.908 INFO:teuthology.orchestra.run.smithi002.stdout: pg 4.14 is active+undersized+degraded, acting [3,7] 2024-09-17T00:25:53.908 INFO:teuthology.orchestra.run.smithi002.stdout: pg 4.1f is active+undersized+degraded, acting [6,5] 2024-09-17T00:25:53.908 INFO:teuthology.orchestra.run.smithi002.stdout: pg 5.1d is active+undersized+degraded, acting [4,0] 2024-09-17T00:25:53.908 INFO:teuthology.orchestra.run.smithi002.stdout: pg 6.1a is active+undersized+degraded, acting [4,5] 2024-09-17T00:25:54.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:25:54 smithi002 ceph-mon[66732]: from='client.44362 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:25:54.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:25:54 smithi002 ceph-mon[66732]: from='client.34467 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:25:54.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:25:54 smithi002 ceph-mon[66732]: from='client.? 172.21.15.2:0/3638681686' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T00:25:54.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:25:54 smithi002 ceph-mon[63804]: from='client.44362 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:25:54.480 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:25:54 smithi002 ceph-mon[63804]: from='client.34467 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:25:54.480 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:25:54 smithi002 ceph-mon[63804]: from='client.? 172.21.15.2:0/3638681686' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T00:25:54.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:25:54 smithi099 ceph-mon[56712]: from='client.44362 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:25:54.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:25:54 smithi099 ceph-mon[56712]: from='client.34467 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:25:54.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:25:54 smithi099 ceph-mon[56712]: from='client.? 172.21.15.2:0/3638681686' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T00:25:55.436 INFO:journalctl@ceph.osd.1.smithi002.stdout:Sep 17 00:25:55 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-1[74140]: 2024-09-17T00:25:55.400+0000 7fd008b56640 -1 osd.1 0 read_superblock omap replica is missing. 2024-09-17T00:25:55.436 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:25:55 smithi002 ceph-mon[66732]: pgmap v85: 161 pgs: 40 active+undersized, 23 active+undersized+degraded, 98 active+clean; 585 KiB data, 121 MiB used, 715 GiB / 715 GiB avail; 104/726 objects degraded (14.325%) 2024-09-17T00:25:55.437 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:25:55 smithi002 ceph-mon[63804]: pgmap v85: 161 pgs: 40 active+undersized, 23 active+undersized+degraded, 98 active+clean; 585 KiB data, 121 MiB used, 715 GiB / 715 GiB avail; 104/726 objects degraded (14.325%) 2024-09-17T00:25:55.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:25:55 smithi099 ceph-mon[56712]: pgmap v85: 161 pgs: 40 active+undersized, 23 active+undersized+degraded, 98 active+clean; 585 KiB data, 121 MiB used, 715 GiB / 715 GiB avail; 104/726 objects degraded (14.325%) 2024-09-17T00:25:55.728 INFO:journalctl@ceph.osd.1.smithi002.stdout:Sep 17 00:25:55 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-1[74140]: 2024-09-17T00:25:55.438+0000 7fd008b56640 -1 osd.1 117 log_to_monitors true 2024-09-17T00:25:56.691 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:25:56 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: ::ffff:172.21.15.99 - - [17/Sep/2024:00:25:56] "GET /metrics HTTP/1.1" 200 38325 "" "Prometheus/2.43.0" 2024-09-17T00:25:56.691 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:25:56 smithi002 ceph-mon[66732]: from='client.44332 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:25:56.691 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:25:56 smithi002 ceph-mon[66732]: from='osd.1 [v2:172.21.15.2:6810/3699540644,v1:172.21.15.2:6811/3699540644]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-17T00:25:56.692 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:25:56 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:25:56.692 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:25:56 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:25:56.692 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:25:56 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:25:56.692 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:25:56 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:25:56.692 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:25:56 smithi002 ceph-mon[63804]: from='client.44332 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:25:56.692 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:25:56 smithi002 ceph-mon[63804]: from='osd.1 [v2:172.21.15.2:6810/3699540644,v1:172.21.15.2:6811/3699540644]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-17T00:25:56.692 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:25:56 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:25:56.692 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:25:56 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:25:56.692 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:25:56 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:25:56.692 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:25:56 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:25:56.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:25:56 smithi099 ceph-mon[56712]: from='client.44332 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:25:56.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:25:56 smithi099 ceph-mon[56712]: from='osd.1 [v2:172.21.15.2:6810/3699540644,v1:172.21.15.2:6811/3699540644]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-17T00:25:56.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:25:56 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:25:56.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:25:56 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:25:56.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:25:56 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:25:56.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:25:56 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:25:56.979 INFO:journalctl@ceph.osd.1.smithi002.stdout:Sep 17 00:25:56 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-1[74140]: 2024-09-17T00:25:56.693+0000 7fcffc8f9640 -1 osd.1 117 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T00:25:57.617 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:25:57 smithi002 ceph-mon[63804]: pgmap v86: 161 pgs: 40 active+undersized, 23 active+undersized+degraded, 98 active+clean; 585 KiB data, 121 MiB used, 715 GiB / 715 GiB avail; 104/726 objects degraded (14.325%) 2024-09-17T00:25:57.618 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:25:57 smithi002 ceph-mon[63804]: from='osd.1 [v2:172.21.15.2:6810/3699540644,v1:172.21.15.2:6811/3699540644]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-17T00:25:57.618 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:25:57 smithi002 ceph-mon[63804]: osdmap e120: 8 total, 7 up, 8 in 2024-09-17T00:25:57.618 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:25:57 smithi002 ceph-mon[63804]: from='osd.1 [v2:172.21.15.2:6810/3699540644,v1:172.21.15.2:6811/3699540644]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]: dispatch 2024-09-17T00:25:57.618 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:25:57 smithi002 ceph-mon[63804]: from='osd.1 [v2:172.21.15.2:6810/3699540644,v1:172.21.15.2:6811/3699540644]' entity='osd.1' 2024-09-17T00:25:57.618 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:25:57 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:25:57.618 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:25:57 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T00:25:57.619 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:25:57 smithi002 ceph-mon[66732]: pgmap v86: 161 pgs: 40 active+undersized, 23 active+undersized+degraded, 98 active+clean; 585 KiB data, 121 MiB used, 715 GiB / 715 GiB avail; 104/726 objects degraded (14.325%) 2024-09-17T00:25:57.619 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:25:57 smithi002 ceph-mon[66732]: from='osd.1 [v2:172.21.15.2:6810/3699540644,v1:172.21.15.2:6811/3699540644]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-17T00:25:57.619 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:25:57 smithi002 ceph-mon[66732]: osdmap e120: 8 total, 7 up, 8 in 2024-09-17T00:25:57.619 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:25:57 smithi002 ceph-mon[66732]: from='osd.1 [v2:172.21.15.2:6810/3699540644,v1:172.21.15.2:6811/3699540644]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]: dispatch 2024-09-17T00:25:57.619 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:25:57 smithi002 ceph-mon[66732]: from='osd.1 [v2:172.21.15.2:6810/3699540644,v1:172.21.15.2:6811/3699540644]' entity='osd.1' 2024-09-17T00:25:57.619 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:25:57 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:25:57.619 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:25:57 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T00:25:57.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:25:57 smithi099 ceph-mon[56712]: pgmap v86: 161 pgs: 40 active+undersized, 23 active+undersized+degraded, 98 active+clean; 585 KiB data, 121 MiB used, 715 GiB / 715 GiB avail; 104/726 objects degraded (14.325%) 2024-09-17T00:25:57.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:25:57 smithi099 ceph-mon[56712]: from='osd.1 [v2:172.21.15.2:6810/3699540644,v1:172.21.15.2:6811/3699540644]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-17T00:25:57.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:25:57 smithi099 ceph-mon[56712]: osdmap e120: 8 total, 7 up, 8 in 2024-09-17T00:25:57.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:25:57 smithi099 ceph-mon[56712]: from='osd.1 [v2:172.21.15.2:6810/3699540644,v1:172.21.15.2:6811/3699540644]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]: dispatch 2024-09-17T00:25:57.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:25:57 smithi099 ceph-mon[56712]: from='osd.1 [v2:172.21.15.2:6810/3699540644,v1:172.21.15.2:6811/3699540644]' entity='osd.1' 2024-09-17T00:25:57.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:25:57 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:25:57.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:25:57 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T00:25:58.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:25:58 smithi099 ceph-mon[56712]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-17T00:25:58.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:25:58 smithi099 ceph-mon[56712]: osd.1 [v2:172.21.15.2:6810/3699540644,v1:172.21.15.2:6811/3699540644] boot 2024-09-17T00:25:58.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:25:58 smithi099 ceph-mon[56712]: osdmap e121: 8 total, 8 up, 8 in 2024-09-17T00:25:58.718 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:25:58 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T00:25:58.718 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:25:58 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[52757]: ts=2024-09-17T00:25:58.659Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd name=CephOSDFlapping index=13 msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.1.1-330-gc2ddf972 (c2ddf9721ef254645201cdbabd9c8f49c30a586f) squid (stable)\", cluster=\"3aad27a2-7489-11ef-bceb-c7b262605968\", cluster_addr=\"172.21.15.2\", device_class=\"ssd\", hostname=\"smithi002\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.2\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.2\", device_class=\"ssd\", hostname=\"smithi002\", instance=\"172.21.15.99:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.2\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-17T00:25:58.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:25:58 smithi002 ceph-mon[66732]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-17T00:25:58.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:25:58 smithi002 ceph-mon[66732]: osd.1 [v2:172.21.15.2:6810/3699540644,v1:172.21.15.2:6811/3699540644] boot 2024-09-17T00:25:58.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:25:58 smithi002 ceph-mon[66732]: osdmap e121: 8 total, 8 up, 8 in 2024-09-17T00:25:58.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:25:58 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T00:25:58.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:25:58 smithi002 ceph-mon[63804]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-17T00:25:58.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:25:58 smithi002 ceph-mon[63804]: osd.1 [v2:172.21.15.2:6810/3699540644,v1:172.21.15.2:6811/3699540644] boot 2024-09-17T00:25:58.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:25:58 smithi002 ceph-mon[63804]: osdmap e121: 8 total, 8 up, 8 in 2024-09-17T00:25:58.731 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:25:58 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-17T00:25:59.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:25:59 smithi002 ceph-mon[63804]: pgmap v89: 161 pgs: 38 peering, 17 active+undersized, 8 active+undersized+degraded, 98 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 45/726 objects degraded (6.198%) 2024-09-17T00:25:59.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:25:59 smithi002 ceph-mon[63804]: Health check update: Degraded data redundancy: 45/726 objects degraded (6.198%), 8 pgs degraded (PG_DEGRADED) 2024-09-17T00:25:59.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:25:59 smithi002 ceph-mon[63804]: osdmap e122: 8 total, 8 up, 8 in 2024-09-17T00:25:59.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:25:59 smithi002 ceph-mon[66732]: pgmap v89: 161 pgs: 38 peering, 17 active+undersized, 8 active+undersized+degraded, 98 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 45/726 objects degraded (6.198%) 2024-09-17T00:25:59.480 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:25:59 smithi002 ceph-mon[66732]: Health check update: Degraded data redundancy: 45/726 objects degraded (6.198%), 8 pgs degraded (PG_DEGRADED) 2024-09-17T00:25:59.480 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:25:59 smithi002 ceph-mon[66732]: osdmap e122: 8 total, 8 up, 8 in 2024-09-17T00:25:59.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:25:59 smithi099 ceph-mon[56712]: pgmap v89: 161 pgs: 38 peering, 17 active+undersized, 8 active+undersized+degraded, 98 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 45/726 objects degraded (6.198%) 2024-09-17T00:25:59.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:25:59 smithi099 ceph-mon[56712]: Health check update: Degraded data redundancy: 45/726 objects degraded (6.198%), 8 pgs degraded (PG_DEGRADED) 2024-09-17T00:25:59.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:25:59 smithi099 ceph-mon[56712]: osdmap e122: 8 total, 8 up, 8 in 2024-09-17T00:26:00.831 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:26:00 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:26:00.832 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:26:00 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:26:00.832 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:26:00 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:26:00.832 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:26:00 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:26:00.832 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:26:00 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:26:00.832 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:26:00 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi002.vmouem", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-17T00:26:00.832 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:26:00 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi002.vmouem", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-17T00:26:00.832 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:26:00 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:26:00.832 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:26:00 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:26:00.833 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:26:00 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:26:00.833 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:26:00 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:26:00.833 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:26:00 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:26:00.833 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:26:00 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:26:00.833 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:26:00 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi002.vmouem", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-17T00:26:00.833 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:26:00 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi002.vmouem", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-17T00:26:00.833 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:26:00 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:26:00.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:26:00 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:26:00.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:26:00 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:26:00.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:26:00 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:26:00.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:26:00 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:26:00.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:26:00 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:26:00.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:26:00 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi002.vmouem", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-17T00:26:00.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:26:00 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi002.vmouem", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-17T00:26:00.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:26:00 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:26:01.960 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:26:01 smithi002 ceph-mon[63804]: Reconfiguring iscsi.foo.smithi002.vmouem (dependencies changed)... 2024-09-17T00:26:01.960 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:26:01 smithi002 ceph-mon[63804]: Reconfiguring daemon iscsi.foo.smithi002.vmouem on smithi002 2024-09-17T00:26:01.960 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:26:01 smithi002 ceph-mon[63804]: pgmap v91: 161 pgs: 38 peering, 17 active+undersized, 8 active+undersized+degraded, 98 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 45/726 objects degraded (6.198%) 2024-09-17T00:26:01.961 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:26:01 smithi002 ceph-mon[66732]: Reconfiguring iscsi.foo.smithi002.vmouem (dependencies changed)... 2024-09-17T00:26:01.961 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:26:01 smithi002 ceph-mon[66732]: Reconfiguring daemon iscsi.foo.smithi002.vmouem on smithi002 2024-09-17T00:26:01.961 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:26:01 smithi002 ceph-mon[66732]: pgmap v91: 161 pgs: 38 peering, 17 active+undersized, 8 active+undersized+degraded, 98 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 45/726 objects degraded (6.198%) 2024-09-17T00:26:01.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:26:01 smithi099 ceph-mon[56712]: Reconfiguring iscsi.foo.smithi002.vmouem (dependencies changed)... 2024-09-17T00:26:01.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:26:01 smithi099 ceph-mon[56712]: Reconfiguring daemon iscsi.foo.smithi002.vmouem on smithi002 2024-09-17T00:26:01.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:26:01 smithi099 ceph-mon[56712]: pgmap v91: 161 pgs: 38 peering, 17 active+undersized, 8 active+undersized+degraded, 98 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 45/726 objects degraded (6.198%) 2024-09-17T00:26:02.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:26:02 smithi002 ceph-mon[66732]: pgmap v92: 161 pgs: 38 peering, 123 active+clean; 585 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:26:02.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:26:02 smithi002 ceph-mon[66732]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 45/726 objects degraded (6.198%), 8 pgs degraded) 2024-09-17T00:26:02.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:26:02 smithi002 ceph-mon[66732]: Cluster is now healthy 2024-09-17T00:26:02.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:26:02 smithi002 ceph-mon[63804]: pgmap v92: 161 pgs: 38 peering, 123 active+clean; 585 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:26:02.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:26:02 smithi002 ceph-mon[63804]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 45/726 objects degraded (6.198%), 8 pgs degraded) 2024-09-17T00:26:02.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:26:02 smithi002 ceph-mon[63804]: Cluster is now healthy 2024-09-17T00:26:02.731 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:26:02 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: 2024-09-17T00:26:02.605+0000 7f17e1e2b640 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi002.vmouem ... 2024-09-17T00:26:02.731 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:26:02 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: Creating ceph-iscsi config... 2024-09-17T00:26:02.731 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:26:02 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/iscsi-gateway.cfg 2024-09-17T00:26:02.731 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:26:02 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/tcmu-runner-entrypoint.sh 2024-09-17T00:26:02.731 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:26:02 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: Non-zero exit code 1 from systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem 2024-09-17T00:26:02.732 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:26:02 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: systemctl: stderr Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:26:02.732 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:26:02 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: systemctl: stderr See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:26:02.732 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:26:02 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: Traceback (most recent call last): 2024-09-17T00:26:02.732 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:26:02 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-17T00:26:02.732 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:26:02 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: return _run_code(code, main_globals, None, 2024-09-17T00:26:02.732 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:26:02 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-17T00:26:02.732 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:26:02 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: exec(code, run_globals) 2024-09-17T00:26:02.732 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:26:02 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5579, in 2024-09-17T00:26:02.732 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:26:02 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5567, in main 2024-09-17T00:26:02.732 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:26:02 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3049, in command_deploy_from 2024-09-17T00:26:02.733 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:26:02 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3084, in _common_deploy 2024-09-17T00:26:02.733 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:26:02 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3104, in _deploy_daemon_container 2024-09-17T00:26:02.733 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:26:02 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 1124, in deploy_daemon 2024-09-17T00:26:02.733 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:26:02 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/tmp/tmpnroy8d0w.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-17T00:26:02.733 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:26:02 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: RuntimeError: Failed command: systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem: Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:26:02.733 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:26:02 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:26:02.733 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:26:02 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: Traceback (most recent call last): 2024-09-17T00:26:02.733 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:26:02 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1134, in _check_daemons 2024-09-17T00:26:02.733 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:26:02 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: self.mgr._daemon_action(daemon_spec, action=action) 2024-09-17T00:26:02.733 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:26:02 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2473, in _daemon_action 2024-09-17T00:26:02.733 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:26:02 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: return self.wait_async( 2024-09-17T00:26:02.733 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:26:02 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/usr/share/ceph/mgr/cephadm/module.py", line 796, in wait_async 2024-09-17T00:26:02.734 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:26:02 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: return self.event_loop.get_result(coro, timeout) 2024-09-17T00:26:02.734 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:26:02 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-09-17T00:26:02.734 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:26:02 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: return future.result(timeout) 2024-09-17T00:26:02.734 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:26:02 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/lib64/python3.9/concurrent/futures/_base.py", line 446, in result 2024-09-17T00:26:02.734 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:26:02 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: return self.__get_result() 2024-09-17T00:26:02.734 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:26:02 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/lib64/python3.9/concurrent/futures/_base.py", line 391, in __get_result 2024-09-17T00:26:02.734 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:26:02 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: raise self._exception 2024-09-17T00:26:02.734 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:26:02 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1370, in _create_daemon 2024-09-17T00:26:02.734 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:26:02 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: out, err, code = await self._run_cephadm( 2024-09-17T00:26:02.734 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:26:02 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1713, in _run_cephadm 2024-09-17T00:26:02.734 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:26:02 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: raise OrchestratorError( 2024-09-17T00:26:02.735 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:26:02 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi002.vmouem ... 2024-09-17T00:26:02.735 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:26:02 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: Creating ceph-iscsi config... 2024-09-17T00:26:02.735 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:26:02 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/iscsi-gateway.cfg 2024-09-17T00:26:02.735 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:26:02 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/tcmu-runner-entrypoint.sh 2024-09-17T00:26:02.735 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:26:02 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: Non-zero exit code 1 from systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem 2024-09-17T00:26:02.735 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:26:02 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: systemctl: stderr Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:26:02.735 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:26:02 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: systemctl: stderr See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:26:02.735 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:26:02 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: Traceback (most recent call last): 2024-09-17T00:26:02.735 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:26:02 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-17T00:26:02.735 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:26:02 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: return _run_code(code, main_globals, None, 2024-09-17T00:26:02.735 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:26:02 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-17T00:26:02.736 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:26:02 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: exec(code, run_globals) 2024-09-17T00:26:02.736 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:26:02 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5579, in 2024-09-17T00:26:02.736 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:26:02 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5567, in main 2024-09-17T00:26:02.736 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:26:02 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3049, in command_deploy_from 2024-09-17T00:26:02.736 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:26:02 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3084, in _common_deploy 2024-09-17T00:26:02.736 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:26:02 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3104, in _deploy_daemon_container 2024-09-17T00:26:02.736 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:26:02 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 1124, in deploy_daemon 2024-09-17T00:26:02.736 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:26:02 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/tmp/tmpnroy8d0w.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-17T00:26:02.736 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:26:02 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: RuntimeError: Failed command: systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem: Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:26:02.736 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:26:02 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:26:02.736 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:26:02 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: 2024-09-17T00:26:02.650+0000 7f17efec7640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (6 PGs are or would become offline) 2024-09-17T00:26:02.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:26:02 smithi099 ceph-mon[56712]: pgmap v92: 161 pgs: 38 peering, 123 active+clean; 585 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:26:02.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:26:02 smithi099 ceph-mon[56712]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 45/726 objects degraded (6.198%), 8 pgs degraded) 2024-09-17T00:26:02.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:26:02 smithi099 ceph-mon[56712]: Cluster is now healthy 2024-09-17T00:26:03.967 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:26:03 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[52757]: ts=2024-09-17T00:26:03.506Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes name=CephNodeDiskspaceWarning index=4 msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi002\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi002\", job=\"node\", machine=\"x86_64\", nodename=\"smithi002\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"3aad27a2-7489-11ef-bceb-c7b262605968\", domainname=\"(none)\", instance=\"smithi002\", job=\"node\", machine=\"x86_64\", nodename=\"smithi002\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-17T00:26:03.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:26:03 smithi099 ceph-mon[56712]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi002.vmouem ... 2024-09-17T00:26:03.968 INFO:journalctl@ceph.mon.b.smithi099.stdout: Creating ceph-iscsi config... 2024-09-17T00:26:03.968 INFO:journalctl@ceph.mon.b.smithi099.stdout: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/iscsi-gateway.cfg 2024-09-17T00:26:03.968 INFO:journalctl@ceph.mon.b.smithi099.stdout: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/tcmu-runner-entrypoint.sh 2024-09-17T00:26:03.968 INFO:journalctl@ceph.mon.b.smithi099.stdout: Non-zero exit code 1 from systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem 2024-09-17T00:26:03.968 INFO:journalctl@ceph.mon.b.smithi099.stdout: systemctl: stderr Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:26:03.968 INFO:journalctl@ceph.mon.b.smithi099.stdout: systemctl: stderr See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:26:03.968 INFO:journalctl@ceph.mon.b.smithi099.stdout: Traceback (most recent call last): 2024-09-17T00:26:03.969 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-17T00:26:03.969 INFO:journalctl@ceph.mon.b.smithi099.stdout: return _run_code(code, main_globals, None, 2024-09-17T00:26:03.969 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-17T00:26:03.970 INFO:journalctl@ceph.mon.b.smithi099.stdout: exec(code, run_globals) 2024-09-17T00:26:03.970 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5579, in 2024-09-17T00:26:03.970 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5567, in main 2024-09-17T00:26:03.970 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3049, in command_deploy_from 2024-09-17T00:26:03.970 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3084, in _common_deploy 2024-09-17T00:26:03.970 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3104, in _deploy_daemon_container 2024-09-17T00:26:03.971 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 1124, in deploy_daemon 2024-09-17T00:26:03.971 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-17T00:26:03.971 INFO:journalctl@ceph.mon.b.smithi099.stdout: RuntimeError: Failed command: systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem: Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:26:03.971 INFO:journalctl@ceph.mon.b.smithi099.stdout: See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:26:03.971 INFO:journalctl@ceph.mon.b.smithi099.stdout: Traceback (most recent call last): 2024-09-17T00:26:03.971 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1134, in _check_daemons 2024-09-17T00:26:03.972 INFO:journalctl@ceph.mon.b.smithi099.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-09-17T00:26:03.972 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2473, in _daemon_action 2024-09-17T00:26:03.972 INFO:journalctl@ceph.mon.b.smithi099.stdout: return self.wait_async( 2024-09-17T00:26:03.972 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 796, in wait_async 2024-09-17T00:26:03.972 INFO:journalctl@ceph.mon.b.smithi099.stdout: return self.event_loop.get_result(coro, timeout) 2024-09-17T00:26:03.973 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-09-17T00:26:03.973 INFO:journalctl@ceph.mon.b.smithi099.stdout: return future.result(timeout) 2024-09-17T00:26:03.973 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/lib64/python3.9/concurrent/futures/_base.py", line 446, in result 2024-09-17T00:26:03.973 INFO:journalctl@ceph.mon.b.smithi099.stdout: return self.__get_result() 2024-09-17T00:26:03.973 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/lib64/python3.9/concurrent/futures/_base.py", line 391, in __get_result 2024-09-17T00:26:03.973 INFO:journalctl@ceph.mon.b.smithi099.stdout: raise self._exception 2024-09-17T00:26:03.973 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1370, in _create_daemon 2024-09-17T00:26:03.973 INFO:journalctl@ceph.mon.b.smithi099.stdout: out, err, code = await self._run_cephadm( 2024-09-17T00:26:03.973 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1713, in _run_cephadm 2024-09-17T00:26:03.973 INFO:journalctl@ceph.mon.b.smithi099.stdout: raise OrchestratorError( 2024-09-17T00:26:03.973 INFO:journalctl@ceph.mon.b.smithi099.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi002.vmouem ... 2024-09-17T00:26:03.974 INFO:journalctl@ceph.mon.b.smithi099.stdout: Creating ceph-iscsi config... 2024-09-17T00:26:03.974 INFO:journalctl@ceph.mon.b.smithi099.stdout: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/iscsi-gateway.cfg 2024-09-17T00:26:03.974 INFO:journalctl@ceph.mon.b.smithi099.stdout: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/tcmu-runner-entrypoint.sh 2024-09-17T00:26:03.974 INFO:journalctl@ceph.mon.b.smithi099.stdout: Non-zero exit code 1 from systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem 2024-09-17T00:26:03.974 INFO:journalctl@ceph.mon.b.smithi099.stdout: systemctl: stderr Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:26:03.974 INFO:journalctl@ceph.mon.b.smithi099.stdout: systemctl: stderr See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:26:03.974 INFO:journalctl@ceph.mon.b.smithi099.stdout: Traceback (most recent call last): 2024-09-17T00:26:03.974 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-17T00:26:03.974 INFO:journalctl@ceph.mon.b.smithi099.stdout: return _run_code(code, main_globals, None, 2024-09-17T00:26:03.974 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-17T00:26:03.974 INFO:journalctl@ceph.mon.b.smithi099.stdout: exec(code, run_globals) 2024-09-17T00:26:03.974 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5579, in 2024-09-17T00:26:03.975 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5567, in main 2024-09-17T00:26:03.975 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3049, in command_deploy_from 2024-09-17T00:26:03.975 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3084, in _common_deploy 2024-09-17T00:26:03.975 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3104, in _deploy_daemon_container 2024-09-17T00:26:03.975 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 1124, in deploy_daemon 2024-09-17T00:26:03.975 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-17T00:26:03.976 INFO:journalctl@ceph.mon.b.smithi099.stdout: RuntimeError: Failed command: systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem: Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:26:03.976 INFO:journalctl@ceph.mon.b.smithi099.stdout: See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:26:03.976 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:26:03 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T00:26:03.976 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:26:03 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:26:03.976 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:26:03 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:26:03.976 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:26:03 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:26:03.976 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:26:03 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-17T00:26:03.976 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:26:03 smithi099 ceph-mon[56712]: from='mon.1 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-17T00:26:03.977 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:26:03 smithi099 ceph-mon[56712]: Upgrade: unsafe to stop osd(s) at this time (6 PGs are or would become offline) 2024-09-17T00:26:03.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:26:03 smithi002 ceph-mon[66732]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi002.vmouem ... 2024-09-17T00:26:03.979 INFO:journalctl@ceph.mon.c.smithi002.stdout: Creating ceph-iscsi config... 2024-09-17T00:26:03.979 INFO:journalctl@ceph.mon.c.smithi002.stdout: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/iscsi-gateway.cfg 2024-09-17T00:26:03.979 INFO:journalctl@ceph.mon.c.smithi002.stdout: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/tcmu-runner-entrypoint.sh 2024-09-17T00:26:03.980 INFO:journalctl@ceph.mon.c.smithi002.stdout: Non-zero exit code 1 from systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem 2024-09-17T00:26:03.980 INFO:journalctl@ceph.mon.c.smithi002.stdout: systemctl: stderr Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:26:03.980 INFO:journalctl@ceph.mon.c.smithi002.stdout: systemctl: stderr See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:26:03.980 INFO:journalctl@ceph.mon.c.smithi002.stdout: Traceback (most recent call last): 2024-09-17T00:26:03.980 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-17T00:26:03.980 INFO:journalctl@ceph.mon.c.smithi002.stdout: return _run_code(code, main_globals, None, 2024-09-17T00:26:03.980 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-17T00:26:03.980 INFO:journalctl@ceph.mon.c.smithi002.stdout: exec(code, run_globals) 2024-09-17T00:26:03.980 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5579, in 2024-09-17T00:26:03.980 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5567, in main 2024-09-17T00:26:03.980 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3049, in command_deploy_from 2024-09-17T00:26:03.980 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3084, in _common_deploy 2024-09-17T00:26:03.980 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3104, in _deploy_daemon_container 2024-09-17T00:26:03.981 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 1124, in deploy_daemon 2024-09-17T00:26:03.981 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-17T00:26:03.981 INFO:journalctl@ceph.mon.c.smithi002.stdout: RuntimeError: Failed command: systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem: Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:26:03.981 INFO:journalctl@ceph.mon.c.smithi002.stdout: See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:26:03.981 INFO:journalctl@ceph.mon.c.smithi002.stdout: Traceback (most recent call last): 2024-09-17T00:26:03.981 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1134, in _check_daemons 2024-09-17T00:26:03.981 INFO:journalctl@ceph.mon.c.smithi002.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-09-17T00:26:03.981 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2473, in _daemon_action 2024-09-17T00:26:03.981 INFO:journalctl@ceph.mon.c.smithi002.stdout: return self.wait_async( 2024-09-17T00:26:03.981 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 796, in wait_async 2024-09-17T00:26:03.981 INFO:journalctl@ceph.mon.c.smithi002.stdout: return self.event_loop.get_result(coro, timeout) 2024-09-17T00:26:03.981 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-09-17T00:26:03.981 INFO:journalctl@ceph.mon.c.smithi002.stdout: return future.result(timeout) 2024-09-17T00:26:03.981 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/lib64/python3.9/concurrent/futures/_base.py", line 446, in result 2024-09-17T00:26:03.982 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:26:03 smithi002 ceph-mon[63804]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi002.vmouem ... 2024-09-17T00:26:03.982 INFO:journalctl@ceph.mon.a.smithi002.stdout: Creating ceph-iscsi config... 2024-09-17T00:26:03.982 INFO:journalctl@ceph.mon.a.smithi002.stdout: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/iscsi-gateway.cfg 2024-09-17T00:26:03.982 INFO:journalctl@ceph.mon.a.smithi002.stdout: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/tcmu-runner-entrypoint.sh 2024-09-17T00:26:03.982 INFO:journalctl@ceph.mon.a.smithi002.stdout: Non-zero exit code 1 from systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem 2024-09-17T00:26:03.982 INFO:journalctl@ceph.mon.a.smithi002.stdout: systemctl: stderr Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:26:03.982 INFO:journalctl@ceph.mon.a.smithi002.stdout: systemctl: stderr See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:26:03.983 INFO:journalctl@ceph.mon.a.smithi002.stdout: Traceback (most recent call last): 2024-09-17T00:26:03.983 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-17T00:26:03.983 INFO:journalctl@ceph.mon.a.smithi002.stdout: return _run_code(code, main_globals, None, 2024-09-17T00:26:03.983 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-17T00:26:03.983 INFO:journalctl@ceph.mon.a.smithi002.stdout: exec(code, run_globals) 2024-09-17T00:26:03.983 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5579, in 2024-09-17T00:26:03.983 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5567, in main 2024-09-17T00:26:03.983 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3049, in command_deploy_from 2024-09-17T00:26:03.983 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3084, in _common_deploy 2024-09-17T00:26:03.983 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3104, in _deploy_daemon_container 2024-09-17T00:26:03.983 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 1124, in deploy_daemon 2024-09-17T00:26:03.983 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-17T00:26:03.983 INFO:journalctl@ceph.mon.a.smithi002.stdout: RuntimeError: Failed command: systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem: Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:26:03.983 INFO:journalctl@ceph.mon.a.smithi002.stdout: See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:26:03.983 INFO:journalctl@ceph.mon.a.smithi002.stdout: Traceback (most recent call last): 2024-09-17T00:26:03.984 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1134, in _check_daemons 2024-09-17T00:26:03.984 INFO:journalctl@ceph.mon.a.smithi002.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-09-17T00:26:03.984 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2473, in _daemon_action 2024-09-17T00:26:03.984 INFO:journalctl@ceph.mon.a.smithi002.stdout: return self.wait_async( 2024-09-17T00:26:03.984 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 796, in wait_async 2024-09-17T00:26:03.984 INFO:journalctl@ceph.mon.a.smithi002.stdout: return self.event_loop.get_result(coro, timeout) 2024-09-17T00:26:03.984 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-09-17T00:26:03.984 INFO:journalctl@ceph.mon.a.smithi002.stdout: return future.result(timeout) 2024-09-17T00:26:03.984 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/lib64/python3.9/concurrent/futures/_base.py", line 446, in result 2024-09-17T00:26:03.984 INFO:journalctl@ceph.mon.c.smithi002.stdout: return self.__get_result() 2024-09-17T00:26:03.985 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/lib64/python3.9/concurrent/futures/_base.py", line 391, in __get_result 2024-09-17T00:26:03.985 INFO:journalctl@ceph.mon.c.smithi002.stdout: raise self._exception 2024-09-17T00:26:03.985 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1370, in _create_daemon 2024-09-17T00:26:03.985 INFO:journalctl@ceph.mon.c.smithi002.stdout: out, err, code = await self._run_cephadm( 2024-09-17T00:26:03.985 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1713, in _run_cephadm 2024-09-17T00:26:03.985 INFO:journalctl@ceph.mon.c.smithi002.stdout: raise OrchestratorError( 2024-09-17T00:26:03.985 INFO:journalctl@ceph.mon.c.smithi002.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi002.vmouem ... 2024-09-17T00:26:03.985 INFO:journalctl@ceph.mon.c.smithi002.stdout: Creating ceph-iscsi config... 2024-09-17T00:26:03.985 INFO:journalctl@ceph.mon.c.smithi002.stdout: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/iscsi-gateway.cfg 2024-09-17T00:26:03.985 INFO:journalctl@ceph.mon.c.smithi002.stdout: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/tcmu-runner-entrypoint.sh 2024-09-17T00:26:03.985 INFO:journalctl@ceph.mon.c.smithi002.stdout: Non-zero exit code 1 from systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem 2024-09-17T00:26:03.985 INFO:journalctl@ceph.mon.c.smithi002.stdout: systemctl: stderr Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:26:03.985 INFO:journalctl@ceph.mon.c.smithi002.stdout: systemctl: stderr See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:26:03.985 INFO:journalctl@ceph.mon.c.smithi002.stdout: Traceback (most recent call last): 2024-09-17T00:26:03.986 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-17T00:26:03.986 INFO:journalctl@ceph.mon.c.smithi002.stdout: return _run_code(code, main_globals, None, 2024-09-17T00:26:03.986 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-17T00:26:03.986 INFO:journalctl@ceph.mon.c.smithi002.stdout: exec(code, run_globals) 2024-09-17T00:26:03.986 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5579, in 2024-09-17T00:26:03.986 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5567, in main 2024-09-17T00:26:03.986 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3049, in command_deploy_from 2024-09-17T00:26:03.986 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3084, in _common_deploy 2024-09-17T00:26:03.986 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3104, in _deploy_daemon_container 2024-09-17T00:26:03.986 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 1124, in deploy_daemon 2024-09-17T00:26:03.986 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-17T00:26:03.987 INFO:journalctl@ceph.mon.c.smithi002.stdout: RuntimeError: Failed command: systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem: Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:26:03.987 INFO:journalctl@ceph.mon.c.smithi002.stdout: See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:26:03.987 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:26:03 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T00:26:03.987 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:26:03 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:26:03.987 INFO:journalctl@ceph.mon.a.smithi002.stdout: return self.__get_result() 2024-09-17T00:26:03.987 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/lib64/python3.9/concurrent/futures/_base.py", line 391, in __get_result 2024-09-17T00:26:03.987 INFO:journalctl@ceph.mon.a.smithi002.stdout: raise self._exception 2024-09-17T00:26:03.987 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1370, in _create_daemon 2024-09-17T00:26:03.987 INFO:journalctl@ceph.mon.a.smithi002.stdout: out, err, code = await self._run_cephadm( 2024-09-17T00:26:03.988 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1713, in _run_cephadm 2024-09-17T00:26:03.988 INFO:journalctl@ceph.mon.a.smithi002.stdout: raise OrchestratorError( 2024-09-17T00:26:03.988 INFO:journalctl@ceph.mon.a.smithi002.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi002.vmouem ... 2024-09-17T00:26:03.988 INFO:journalctl@ceph.mon.a.smithi002.stdout: Creating ceph-iscsi config... 2024-09-17T00:26:03.988 INFO:journalctl@ceph.mon.a.smithi002.stdout: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/iscsi-gateway.cfg 2024-09-17T00:26:03.988 INFO:journalctl@ceph.mon.a.smithi002.stdout: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/tcmu-runner-entrypoint.sh 2024-09-17T00:26:03.988 INFO:journalctl@ceph.mon.a.smithi002.stdout: Non-zero exit code 1 from systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem 2024-09-17T00:26:03.988 INFO:journalctl@ceph.mon.a.smithi002.stdout: systemctl: stderr Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:26:03.988 INFO:journalctl@ceph.mon.a.smithi002.stdout: systemctl: stderr See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:26:03.988 INFO:journalctl@ceph.mon.a.smithi002.stdout: Traceback (most recent call last): 2024-09-17T00:26:03.989 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-17T00:26:03.989 INFO:journalctl@ceph.mon.a.smithi002.stdout: return _run_code(code, main_globals, None, 2024-09-17T00:26:03.989 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-17T00:26:03.989 INFO:journalctl@ceph.mon.a.smithi002.stdout: exec(code, run_globals) 2024-09-17T00:26:03.989 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5579, in 2024-09-17T00:26:03.989 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5567, in main 2024-09-17T00:26:03.989 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3049, in command_deploy_from 2024-09-17T00:26:03.989 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3084, in _common_deploy 2024-09-17T00:26:03.989 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3104, in _deploy_daemon_container 2024-09-17T00:26:03.989 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 1124, in deploy_daemon 2024-09-17T00:26:03.989 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-17T00:26:03.989 INFO:journalctl@ceph.mon.a.smithi002.stdout: RuntimeError: Failed command: systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem: Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:26:03.990 INFO:journalctl@ceph.mon.a.smithi002.stdout: See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:26:03.990 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:26:03 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T00:26:03.990 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:26:03 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:26:03.990 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:26:03 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:26:03.990 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:26:03 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:26:03.990 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:26:03 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-17T00:26:03.990 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:26:03 smithi002 ceph-mon[66732]: from='mon.1 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-17T00:26:03.990 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:26:03 smithi002 ceph-mon[66732]: Upgrade: unsafe to stop osd(s) at this time (6 PGs are or would become offline) 2024-09-17T00:26:03.990 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:26:03 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:26:03.991 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:26:03 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:26:03.991 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:26:03 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-17T00:26:03.991 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:26:03 smithi002 ceph-mon[63804]: from='mon.1 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-17T00:26:03.991 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:26:03 smithi002 ceph-mon[63804]: Upgrade: unsafe to stop osd(s) at this time (6 PGs are or would become offline) 2024-09-17T00:26:04.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:26:04 smithi099 ceph-mon[56712]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-17T00:26:04.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:26:04 smithi002 ceph-mon[66732]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-17T00:26:04.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:26:04 smithi002 ceph-mon[63804]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-17T00:26:06.979 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:26:06 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: ::ffff:172.21.15.99 - - [17/Sep/2024:00:26:06] "GET /metrics HTTP/1.1" 200 38269 "" "Prometheus/2.43.0" 2024-09-17T00:26:07.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:26:07 smithi099 ceph-mon[56712]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:26:07.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:26:07 smithi002 ceph-mon[66732]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:26:07.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:26:07 smithi002 ceph-mon[63804]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:26:08.967 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:26:08 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[52757]: ts=2024-09-17T00:26:08.659Z caller=alerting.go:366 level=warn component="rule manager" alert="unsupported value type" msg="Expanding alert template failed" err="error executing template __alert_CephOSDDown: template: __alert_CephOSDDown:1:358: executing \"__alert_CephOSDDown\" at : error calling query: found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.1.1-330-gc2ddf972 (c2ddf9721ef254645201cdbabd9c8f49c30a586f) squid (stable)\", cluster=\"3aad27a2-7489-11ef-bceb-c7b262605968\", cluster_addr=\"172.21.15.2\", device_class=\"ssd\", hostname=\"smithi002\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.2\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.2\", device_class=\"ssd\", hostname=\"smithi002\", instance=\"172.21.15.99:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.2\"}];many-to-many matching not allowed: matching labels must be unique on one side" data="unsupported value type" 2024-09-17T00:26:08.967 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:26:08 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[52757]: ts=2024-09-17T00:26:08.660Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd name=CephOSDFlapping index=13 msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.1.1-330-gc2ddf972 (c2ddf9721ef254645201cdbabd9c8f49c30a586f) squid (stable)\", cluster=\"3aad27a2-7489-11ef-bceb-c7b262605968\", cluster_addr=\"172.21.15.2\", device_class=\"ssd\", hostname=\"smithi002\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.2\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.2\", device_class=\"ssd\", hostname=\"smithi002\", instance=\"172.21.15.99:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.2\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-17T00:26:09.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:26:09 smithi099 ceph-mon[56712]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-09-17T00:26:09.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:26:09 smithi002 ceph-mon[63804]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-09-17T00:26:09.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:26:09 smithi002 ceph-mon[66732]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-09-17T00:26:11.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:26:11 smithi099 ceph-mon[56712]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 446 B/s rd, 0 op/s 2024-09-17T00:26:11.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:26:11 smithi002 ceph-mon[66732]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 446 B/s rd, 0 op/s 2024-09-17T00:26:11.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:26:11 smithi002 ceph-mon[63804]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 446 B/s rd, 0 op/s 2024-09-17T00:26:13.140 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:26:12 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:26:13.140 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:26:12 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T00:26:13.140 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:26:12 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:26:13.140 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:26:12 smithi002 ceph-mon[66732]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-09-17T00:26:13.141 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:26:12 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:26:13.141 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:26:12 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T00:26:13.141 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:26:12 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:26:13.141 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:26:12 smithi002 ceph-mon[63804]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-09-17T00:26:13.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:26:12 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:26:13.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:26:12 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T00:26:13.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:26:12 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:26:13.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:26:12 smithi099 ceph-mon[56712]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-09-17T00:26:13.967 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:26:13 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[52757]: ts=2024-09-17T00:26:13.506Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes name=CephNodeDiskspaceWarning index=4 msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi002\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi002\", job=\"node\", machine=\"x86_64\", nodename=\"smithi002\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"3aad27a2-7489-11ef-bceb-c7b262605968\", domainname=\"(none)\", instance=\"smithi002\", job=\"node\", machine=\"x86_64\", nodename=\"smithi002\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-17T00:26:15.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:26:15 smithi099 ceph-mon[56712]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-09-17T00:26:15.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:26:15 smithi099 ceph-mon[56712]: from='client.? 172.21.15.2:0/1948484199' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-17T00:26:15.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:26:15 smithi002 ceph-mon[66732]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-09-17T00:26:15.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:26:15 smithi002 ceph-mon[66732]: from='client.? 172.21.15.2:0/1948484199' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-17T00:26:15.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:26:15 smithi002 ceph-mon[63804]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-09-17T00:26:15.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:26:15 smithi002 ceph-mon[63804]: from='client.? 172.21.15.2:0/1948484199' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-17T00:26:16.979 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:26:16 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: ::ffff:172.21.15.99 - - [17/Sep/2024:00:26:16] "GET /metrics HTTP/1.1" 200 38423 "" "Prometheus/2.43.0" 2024-09-17T00:26:17.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:26:17 smithi099 ceph-mon[56712]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 141 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:26:17.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:26:17 smithi002 ceph-mon[63804]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 141 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:26:17.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:26:17 smithi002 ceph-mon[66732]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 141 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:26:18.390 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:26:18 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-17T00:26:18.390 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:26:18 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-17T00:26:18.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:26:18 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-17T00:26:18.967 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:26:18 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[52757]: ts=2024-09-17T00:26:18.659Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd name=CephOSDFlapping index=13 msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.1.1-330-gc2ddf972 (c2ddf9721ef254645201cdbabd9c8f49c30a586f) squid (stable)\", cluster=\"3aad27a2-7489-11ef-bceb-c7b262605968\", cluster_addr=\"172.21.15.2\", device_class=\"ssd\", hostname=\"smithi002\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.2\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.2\", device_class=\"ssd\", hostname=\"smithi002\", instance=\"172.21.15.99:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.2\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-17T00:26:19.299 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:26:19 smithi002 ceph-mon[66732]: from='mon.1 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-17T00:26:19.299 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:26:19 smithi002 ceph-mon[66732]: Upgrade: osd.2 is safe to restart 2024-09-17T00:26:19.299 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:26:19 smithi002 ceph-mon[66732]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-09-17T00:26:19.299 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:26:19 smithi002 ceph-mon[63804]: from='mon.1 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-17T00:26:19.299 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:26:19 smithi002 ceph-mon[63804]: Upgrade: osd.2 is safe to restart 2024-09-17T00:26:19.299 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:26:19 smithi002 ceph-mon[63804]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-09-17T00:26:19.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:26:19 smithi099 ceph-mon[56712]: from='mon.1 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-17T00:26:19.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:26:19 smithi099 ceph-mon[56712]: Upgrade: osd.2 is safe to restart 2024-09-17T00:26:19.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:26:19 smithi099 ceph-mon[56712]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-09-17T00:26:20.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:26:20 smithi099 ceph-mon[56712]: Upgrade: Updating osd.2 2024-09-17T00:26:20.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:26:20 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:26:20.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:26:20 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-09-17T00:26:20.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:26:20 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:26:20.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:26:20 smithi099 ceph-mon[56712]: Deploying daemon osd.2 on smithi002 2024-09-17T00:26:20.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:26:20 smithi002 ceph-mon[63804]: Upgrade: Updating osd.2 2024-09-17T00:26:20.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:26:20 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:26:20.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:26:20 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-09-17T00:26:20.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:26:20 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:26:20.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:26:20 smithi002 ceph-mon[63804]: Deploying daemon osd.2 on smithi002 2024-09-17T00:26:20.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:26:20 smithi002 ceph-mon[66732]: Upgrade: Updating osd.2 2024-09-17T00:26:20.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:26:20 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:26:20.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:26:20 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-09-17T00:26:20.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:26:20 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:26:20.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:26:20 smithi002 ceph-mon[66732]: Deploying daemon osd.2 on smithi002 2024-09-17T00:26:21.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:26:21 smithi099 ceph-mon[56712]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-09-17T00:26:21.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:26:21 smithi002 ceph-mon[63804]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-09-17T00:26:21.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:26:21 smithi002 ceph-mon[66732]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-09-17T00:26:22.729 INFO:journalctl@ceph.osd.2.smithi002.stdout:Sep 17 00:26:22 smithi002 systemd[1]: Stopping Ceph osd.2 for 3aad27a2-7489-11ef-bceb-c7b262605968... 2024-09-17T00:26:23.229 INFO:journalctl@ceph.osd.2.smithi002.stdout:Sep 17 00:26:22 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-2[36852]: 2024-09-17T00:26:22.749+0000 7fea9e3c7700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-17T00:26:23.229 INFO:journalctl@ceph.osd.2.smithi002.stdout:Sep 17 00:26:22 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-2[36852]: 2024-09-17T00:26:22.749+0000 7fea9e3c7700 -1 osd.2 122 *** Got signal Terminated *** 2024-09-17T00:26:23.229 INFO:journalctl@ceph.osd.2.smithi002.stdout:Sep 17 00:26:22 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-2[36852]: 2024-09-17T00:26:22.749+0000 7fea9e3c7700 -1 osd.2 122 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-17T00:26:23.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:26:23 smithi099 ceph-mon[56712]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-09-17T00:26:23.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:26:23 smithi099 ceph-mon[56712]: osd.2 marked itself down and dead 2024-09-17T00:26:23.717 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:26:23 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[52757]: ts=2024-09-17T00:26:23.506Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes name=CephNodeDiskspaceWarning index=4 msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi002\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi002\", job=\"node\", machine=\"x86_64\", nodename=\"smithi002\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"3aad27a2-7489-11ef-bceb-c7b262605968\", domainname=\"(none)\", instance=\"smithi002\", job=\"node\", machine=\"x86_64\", nodename=\"smithi002\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-17T00:26:23.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:26:23 smithi002 ceph-mon[66732]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-09-17T00:26:23.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:26:23 smithi002 ceph-mon[66732]: osd.2 marked itself down and dead 2024-09-17T00:26:23.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:26:23 smithi002 ceph-mon[63804]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-09-17T00:26:23.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:26:23 smithi002 ceph-mon[63804]: osd.2 marked itself down and dead 2024-09-17T00:26:24.064 INFO:journalctl@ceph.osd.2.smithi002.stdout:Sep 17 00:26:23 smithi002 podman[76599]: 2024-09-17 00:26:23.803608963 +0000 UTC m=+1.172407739 container died eafddacb955e5de7b0f33bcd4216dd6143d61859b307423df7b2ec49bcd71c99 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-2, io.k8s.display-name=CentOS Stream 8, io.buildah.version=1.19.8, distribution-scope=public, maintainer=Guillaume Abrioux , RELEASE=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.expose-services=, com.redhat.component=centos-stream-container, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, release=754, name=centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base centos centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., version=8, build-date=2022-05-03T08:36:31.336870, CEPH_POINT_RELEASE=-17.2.0, GIT_BRANCH=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vendor=Red Hat, Inc., vcs-type=git, ceph=True) 2024-09-17T00:26:24.323 INFO:journalctl@ceph.osd.2.smithi002.stdout:Sep 17 00:26:24 smithi002 podman[76599]: 2024-09-17 00:26:24.166095218 +0000 UTC m=+1.534893999 container cleanup eafddacb955e5de7b0f33bcd4216dd6143d61859b307423df7b2ec49bcd71c99 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-2, io.openshift.expose-services=, release=754, architecture=x86_64, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.tags=base centos centos-stream, maintainer=Guillaume Abrioux , summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_CLEAN=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, version=8, build-date=2022-05-03T08:36:31.336870, ceph=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, CEPH_POINT_RELEASE=-17.2.0, io.k8s.display-name=CentOS Stream 8, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., distribution-scope=public, RELEASE=HEAD, name=centos-stream, vendor=Red Hat, Inc., io.buildah.version=1.19.8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vcs-type=git, com.redhat.component=centos-stream-container, GIT_BRANCH=HEAD) 2024-09-17T00:26:24.323 INFO:journalctl@ceph.osd.2.smithi002.stdout:Sep 17 00:26:24 smithi002 bash[76599]: ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-2 2024-09-17T00:26:24.324 INFO:teuthology.orchestra.run.smithi002.stdout:true 2024-09-17T00:26:24.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:26:24 smithi099 ceph-mon[56712]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-17T00:26:24.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:26:24 smithi099 ceph-mon[56712]: osdmap e123: 8 total, 7 up, 8 in 2024-09-17T00:26:24.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:26:24 smithi002 ceph-mon[66732]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-17T00:26:24.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:26:24 smithi002 ceph-mon[66732]: osdmap e123: 8 total, 7 up, 8 in 2024-09-17T00:26:24.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:26:24 smithi002 ceph-mon[63804]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-17T00:26:24.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:26:24 smithi002 ceph-mon[63804]: osdmap e123: 8 total, 7 up, 8 in 2024-09-17T00:26:24.730 INFO:journalctl@ceph.osd.2.smithi002.stdout:Sep 17 00:26:24 smithi002 podman[76616]: 2024-09-17 00:26:24.467400503 +0000 UTC m=+0.664336479 container remove eafddacb955e5de7b0f33bcd4216dd6143d61859b307423df7b2ec49bcd71c99 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-2, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, distribution-scope=public, build-date=2022-05-03T08:36:31.336870, RELEASE=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vendor=Red Hat, Inc., ceph=True, com.redhat.component=centos-stream-container, io.buildah.version=1.19.8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base centos centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, maintainer=Guillaume Abrioux , architecture=x86_64, CEPH_POINT_RELEASE=-17.2.0, release=754, GIT_CLEAN=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.display-name=CentOS Stream 8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-type=git, io.openshift.expose-services=, GIT_REPO=https://github.com/ceph/ceph-container.git, version=8, name=centos-stream, GIT_BRANCH=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754) 2024-09-17T00:26:25.078 INFO:teuthology.orchestra.run.smithi002.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-17T00:26:25.078 INFO:teuthology.orchestra.run.smithi002.stdout:alertmanager.a smithi002 *:9093,9094 running (4m) 29s ago 9m 26.3M - 0.25.0 c8568f914cd2 a21e13b3d677 2024-09-17T00:26:25.078 INFO:teuthology.orchestra.run.smithi002.stdout:grafana.a smithi099 *:3000 running (3m) 117s ago 8m 46.9M - 8.3.5 dad864ee21e9 bafb824a3c08 2024-09-17T00:26:25.078 INFO:teuthology.orchestra.run.smithi002.stdout:iscsi.foo.smithi002.vmouem smithi002 running (50s) 29s ago 8m 50.4M - 3.5 e1d6a67b021e ebed893170ce 2024-09-17T00:26:25.078 INFO:teuthology.orchestra.run.smithi002.stdout:mgr.x smithi099 *:8443,9283,8765 running (3m) 117s ago 13m 439M - 19.1.1-330-gc2ddf972 ebb0615149de 6aa2ccef9ca0 2024-09-17T00:26:25.078 INFO:teuthology.orchestra.run.smithi002.stdout:mgr.y smithi002 *:8443,9283,8765 running (3m) 29s ago 15m 502M - 19.1.1-330-gc2ddf972 ebb0615149de 8838897e8449 2024-09-17T00:26:25.078 INFO:teuthology.orchestra.run.smithi002.stdout:mon.a smithi002 running (2m) 29s ago 15m 49.2M 2048M 19.1.1-330-gc2ddf972 ebb0615149de a035def53ef6 2024-09-17T00:26:25.078 INFO:teuthology.orchestra.run.smithi002.stdout:mon.b smithi099 running (2m) 117s ago 14m 17.5M 2048M 19.1.1-330-gc2ddf972 ebb0615149de aab3ab73a850 2024-09-17T00:26:25.078 INFO:teuthology.orchestra.run.smithi002.stdout:mon.c smithi002 running (2m) 29s ago 14m 41.8M 2048M 19.1.1-330-gc2ddf972 ebb0615149de 8a510255ae66 2024-09-17T00:26:25.078 INFO:teuthology.orchestra.run.smithi002.stdout:node-exporter.a smithi002 *:9100 running (4m) 29s ago 9m 15.1M - 1.5.0 0da6a335fe13 1030ae01bb32 2024-09-17T00:26:25.079 INFO:teuthology.orchestra.run.smithi002.stdout:node-exporter.b smithi099 *:9100 running (3m) 117s ago 9m 16.8M - 1.5.0 0da6a335fe13 01fcd35dca8a 2024-09-17T00:26:25.079 INFO:teuthology.orchestra.run.smithi002.stdout:osd.0 smithi002 running (74s) 29s ago 13m 43.9M 2737M 19.1.1-330-gc2ddf972 ebb0615149de 8345719cb6f0 2024-09-17T00:26:25.079 INFO:teuthology.orchestra.run.smithi002.stdout:osd.1 smithi002 running (35s) 29s ago 13m 12.4M 2737M 19.1.1-330-gc2ddf972 ebb0615149de 3dbc23d57c53 2024-09-17T00:26:25.079 INFO:teuthology.orchestra.run.smithi002.stdout:osd.2 smithi002 running (12m) 29s ago 12m 55.0M 2737M 17.2.0 e1d6a67b021e eafddacb955e 2024-09-17T00:26:25.079 INFO:teuthology.orchestra.run.smithi002.stdout:osd.3 smithi002 running (12m) 29s ago 12m 57.7M 2737M 17.2.0 e1d6a67b021e fd99d3764412 2024-09-17T00:26:25.079 INFO:teuthology.orchestra.run.smithi002.stdout:osd.4 smithi099 running (11m) 117s ago 11m 55.4M 2992M 17.2.0 e1d6a67b021e 0af1827b658d 2024-09-17T00:26:25.079 INFO:teuthology.orchestra.run.smithi002.stdout:osd.5 smithi099 running (11m) 117s ago 11m 54.2M 2992M 17.2.0 e1d6a67b021e 4f7ef3aefc58 2024-09-17T00:26:25.079 INFO:teuthology.orchestra.run.smithi002.stdout:osd.6 smithi099 running (11m) 117s ago 11m 51.0M 2992M 17.2.0 e1d6a67b021e 3213f5e6952a 2024-09-17T00:26:25.079 INFO:teuthology.orchestra.run.smithi002.stdout:osd.7 smithi099 running (10m) 117s ago 10m 56.6M 2992M 17.2.0 e1d6a67b021e 63be3a1ed0a1 2024-09-17T00:26:25.079 INFO:teuthology.orchestra.run.smithi002.stdout:prometheus.a smithi099 *:9095 running (3m) 117s ago 9m 50.1M - 2.43.0 a07b618ecd1d bf32c2ae11dc 2024-09-17T00:26:25.079 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.foo.smithi002.uwdhcs smithi002 *:8000 running (8m) 29s ago 8m 75.8M - 17.2.0 e1d6a67b021e 9c35813f459a 2024-09-17T00:26:25.079 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.foo.smithi099.utzfap smithi099 *:8000 running (8m) 117s ago 8m 76.0M - 17.2.0 e1d6a67b021e 34ec989160a6 2024-09-17T00:26:25.079 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.smpl.smithi002.rqnxii smithi002 *:80 running (8m) 29s ago 8m 74.1M - 17.2.0 e1d6a67b021e 0e180d0afb99 2024-09-17T00:26:25.080 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.smpl.smithi099.nzvkpm smithi099 *:80 running (8m) 117s ago 8m 76.5M - 17.2.0 e1d6a67b021e 6a068e82ac72 2024-09-17T00:26:25.142 INFO:journalctl@ceph.osd.2.smithi002.stdout:Sep 17 00:26:24 smithi002 podman[76754]: 2024-09-17 00:26:24.747557764 +0000 UTC m=+0.017692719 image pull ebb0615149def4a3179c82c0cc47af75808300abb9f1142788e5e605a7ac0e5b quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f 2024-09-17T00:26:25.142 INFO:journalctl@ceph.osd.2.smithi002.stdout:Sep 17 00:26:24 smithi002 podman[76754]: 2024-09-17 00:26:24.868272169 +0000 UTC m=+0.138407114 container create 52e647b08beeafbaf54f3520d821bebc2df4d9d465c572083b008935e2d8a667 (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-2-deactivate, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-c2ddf97, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-09-17T00:26:25.143 INFO:journalctl@ceph.osd.2.smithi002.stdout:Sep 17 00:26:25 smithi002 podman[76754]: 2024-09-17 00:26:25.079059144 +0000 UTC m=+0.349194099 container init 52e647b08beeafbaf54f3520d821bebc2df4d9d465c572083b008935e2d8a667 (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-2-deactivate, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=squid-c2ddf97, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_CLEAN=True) 2024-09-17T00:26:25.143 INFO:journalctl@ceph.osd.2.smithi002.stdout:Sep 17 00:26:25 smithi002 podman[76754]: 2024-09-17 00:26:25.083291725 +0000 UTC m=+0.353426677 container start 52e647b08beeafbaf54f3520d821bebc2df4d9d465c572083b008935e2d8a667 (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-2-deactivate, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=squid-c2ddf97, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-17T00:26:25.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:26:25 smithi002 ceph-mon[66732]: pgmap v104: 161 pgs: 12 stale+active+clean, 149 active+clean; 585 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-09-17T00:26:25.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:26:25 smithi002 ceph-mon[66732]: from='client.44398 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:26:25.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:26:25 smithi002 ceph-mon[66732]: from='client.44410 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:26:25.480 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:26:25 smithi002 ceph-mon[66732]: osdmap e124: 8 total, 7 up, 8 in 2024-09-17T00:26:25.480 INFO:journalctl@ceph.osd.2.smithi002.stdout:Sep 17 00:26:25 smithi002 podman[76754]: 2024-09-17 00:26:25.145698047 +0000 UTC m=+0.415832999 container attach 52e647b08beeafbaf54f3520d821bebc2df4d9d465c572083b008935e2d8a667 (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, io.buildah.version=1.37.2, RELEASE=squid-c2ddf97, org.label-schema.build-date=20240731, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-09-17T00:26:25.480 INFO:journalctl@ceph.osd.2.smithi002.stdout:Sep 17 00:26:25 smithi002 podman[76754]: 2024-09-17 00:26:25.297018202 +0000 UTC m=+0.567153151 container died 52e647b08beeafbaf54f3520d821bebc2df4d9d465c572083b008935e2d8a667 (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-2-deactivate, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=squid-c2ddf97, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-09-17T00:26:25.480 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:26:25 smithi002 ceph-mon[63804]: pgmap v104: 161 pgs: 12 stale+active+clean, 149 active+clean; 585 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-09-17T00:26:25.480 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:26:25 smithi002 ceph-mon[63804]: from='client.44398 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:26:25.480 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:26:25 smithi002 ceph-mon[63804]: from='client.44410 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:26:25.480 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:26:25 smithi002 ceph-mon[63804]: osdmap e124: 8 total, 7 up, 8 in 2024-09-17T00:26:25.522 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-09-17T00:26:25.522 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": { 2024-09-17T00:26:25.522 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 19.1.1-330-gc2ddf972 (c2ddf9721ef254645201cdbabd9c8f49c30a586f) squid (stable)": 3 2024-09-17T00:26:25.522 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-09-17T00:26:25.522 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": { 2024-09-17T00:26:25.522 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 19.1.1-330-gc2ddf972 (c2ddf9721ef254645201cdbabd9c8f49c30a586f) squid (stable)": 2 2024-09-17T00:26:25.522 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-09-17T00:26:25.522 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": { 2024-09-17T00:26:25.522 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 5, 2024-09-17T00:26:25.522 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 19.1.1-330-gc2ddf972 (c2ddf9721ef254645201cdbabd9c8f49c30a586f) squid (stable)": 2 2024-09-17T00:26:25.523 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-09-17T00:26:25.523 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw": { 2024-09-17T00:26:25.523 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-09-17T00:26:25.523 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-09-17T00:26:25.523 INFO:teuthology.orchestra.run.smithi002.stdout: "overall": { 2024-09-17T00:26:25.523 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 9, 2024-09-17T00:26:25.523 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 19.1.1-330-gc2ddf972 (c2ddf9721ef254645201cdbabd9c8f49c30a586f) squid (stable)": 7 2024-09-17T00:26:25.523 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-09-17T00:26:25.523 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-09-17T00:26:25.618 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:26:25 smithi099 ceph-mon[56712]: pgmap v104: 161 pgs: 12 stale+active+clean, 149 active+clean; 585 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-09-17T00:26:25.618 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:26:25 smithi099 ceph-mon[56712]: from='client.44398 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:26:25.618 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:26:25 smithi099 ceph-mon[56712]: from='client.44410 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:26:25.618 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:26:25 smithi099 ceph-mon[56712]: osdmap e124: 8 total, 7 up, 8 in 2024-09-17T00:26:25.874 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-09-17T00:26:25.874 INFO:teuthology.orchestra.run.smithi002.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f", 2024-09-17T00:26:25.874 INFO:teuthology.orchestra.run.smithi002.stdout: "in_progress": true, 2024-09-17T00:26:25.874 INFO:teuthology.orchestra.run.smithi002.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-17T00:26:25.874 INFO:teuthology.orchestra.run.smithi002.stdout: "services_complete": [ 2024-09-17T00:26:25.874 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr", 2024-09-17T00:26:25.874 INFO:teuthology.orchestra.run.smithi002.stdout: "mon" 2024-09-17T00:26:25.874 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-09-17T00:26:25.874 INFO:teuthology.orchestra.run.smithi002.stdout: "progress": "7/23 daemons upgraded", 2024-09-17T00:26:25.874 INFO:teuthology.orchestra.run.smithi002.stdout: "message": "Currently upgrading osd daemons", 2024-09-17T00:26:25.874 INFO:teuthology.orchestra.run.smithi002.stdout: "is_paused": false 2024-09-17T00:26:25.874 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-09-17T00:26:26.206 INFO:journalctl@ceph.osd.2.smithi002.stdout:Sep 17 00:26:25 smithi002 podman[76822]: 2024-09-17 00:26:25.978390002 +0000 UTC m=+0.670231598 container remove 52e647b08beeafbaf54f3520d821bebc2df4d9d465c572083b008935e2d8a667 (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-2-deactivate, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=squid-c2ddf97, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-09-17T00:26:26.207 INFO:journalctl@ceph.osd.2.smithi002.stdout:Sep 17 00:26:25 smithi002 systemd[1]: ceph-3aad27a2-7489-11ef-bceb-c7b262605968@osd.2.service: Deactivated successfully. 2024-09-17T00:26:26.207 INFO:journalctl@ceph.osd.2.smithi002.stdout:Sep 17 00:26:25 smithi002 systemd[1]: Stopped Ceph osd.2 for 3aad27a2-7489-11ef-bceb-c7b262605968. 2024-09-17T00:26:26.207 INFO:journalctl@ceph.osd.2.smithi002.stdout:Sep 17 00:26:25 smithi002 systemd[1]: ceph-3aad27a2-7489-11ef-bceb-c7b262605968@osd.2.service: Consumed 7.390s CPU time. 2024-09-17T00:26:26.322 INFO:teuthology.orchestra.run.smithi002.stdout:HEALTH_WARN 1 osds down 2024-09-17T00:26:26.323 INFO:teuthology.orchestra.run.smithi002.stdout:[WRN] OSD_DOWN: 1 osds down 2024-09-17T00:26:26.323 INFO:teuthology.orchestra.run.smithi002.stdout: osd.2 (root=default,host=smithi002) is down 2024-09-17T00:26:26.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:26:26 smithi002 ceph-mon[66732]: from='client.44416 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:26:26.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:26:26 smithi002 ceph-mon[66732]: from='client.34518 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:26:26.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:26:26 smithi002 ceph-mon[66732]: from='client.? 172.21.15.2:0/506340030' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:26:26.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:26:26 smithi002 ceph-mon[66732]: from='client.? 172.21.15.2:0/898213741' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T00:26:26.480 INFO:journalctl@ceph.osd.2.smithi002.stdout:Sep 17 00:26:26 smithi002 systemd[1]: Starting Ceph osd.2 for 3aad27a2-7489-11ef-bceb-c7b262605968... 2024-09-17T00:26:26.480 INFO:journalctl@ceph.osd.2.smithi002.stdout:Sep 17 00:26:26 smithi002 podman[76966]: 2024-09-17 00:26:26.336624034 +0000 UTC m=+0.017364529 image pull ebb0615149def4a3179c82c0cc47af75808300abb9f1142788e5e605a7ac0e5b quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f 2024-09-17T00:26:26.480 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:26:26 smithi002 ceph-mon[63804]: from='client.44416 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:26:26.480 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:26:26 smithi002 ceph-mon[63804]: from='client.34518 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:26:26.480 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:26:26 smithi002 ceph-mon[63804]: from='client.? 172.21.15.2:0/506340030' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:26:26.480 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:26:26 smithi002 ceph-mon[63804]: from='client.? 172.21.15.2:0/898213741' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T00:26:26.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:26:26 smithi099 ceph-mon[56712]: from='client.44416 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:26:26.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:26:26 smithi099 ceph-mon[56712]: from='client.34518 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:26:26.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:26:26 smithi099 ceph-mon[56712]: from='client.? 172.21.15.2:0/506340030' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:26:26.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:26:26 smithi099 ceph-mon[56712]: from='client.? 172.21.15.2:0/898213741' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T00:26:26.759 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:26:26 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: ::ffff:172.21.15.99 - - [17/Sep/2024:00:26:26] "GET /metrics HTTP/1.1" 200 38423 "" "Prometheus/2.43.0" 2024-09-17T00:26:26.759 INFO:journalctl@ceph.osd.2.smithi002.stdout:Sep 17 00:26:26 smithi002 podman[76966]: 2024-09-17 00:26:26.499301267 +0000 UTC m=+0.180041793 container create efc318ad5f0f352878ce8c512f1077841379192582668039451c5831b224c4a2 (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-2-activate, CEPH_POINT_RELEASE=, RELEASE=squid-c2ddf97, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-09-17T00:26:27.229 INFO:journalctl@ceph.osd.2.smithi002.stdout:Sep 17 00:26:26 smithi002 podman[76966]: 2024-09-17 00:26:26.762601889 +0000 UTC m=+0.443342387 container init efc318ad5f0f352878ce8c512f1077841379192582668039451c5831b224c4a2 (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-2-activate, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, ceph=True, org.label-schema.license=GPLv2, RELEASE=squid-c2ddf97, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD) 2024-09-17T00:26:27.229 INFO:journalctl@ceph.osd.2.smithi002.stdout:Sep 17 00:26:26 smithi002 podman[76966]: 2024-09-17 00:26:26.768125465 +0000 UTC m=+0.448865952 container start efc318ad5f0f352878ce8c512f1077841379192582668039451c5831b224c4a2 (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-2-activate, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, RELEASE=squid-c2ddf97, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-17T00:26:27.229 INFO:journalctl@ceph.osd.2.smithi002.stdout:Sep 17 00:26:26 smithi002 podman[76966]: 2024-09-17 00:26:26.870762368 +0000 UTC m=+0.551502863 container attach efc318ad5f0f352878ce8c512f1077841379192582668039451c5831b224c4a2 (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-2-activate, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , RELEASE=squid-c2ddf97, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD) 2024-09-17T00:26:27.230 INFO:journalctl@ceph.osd.2.smithi002.stdout:Sep 17 00:26:26 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-2-activate[76984]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-17T00:26:27.230 INFO:journalctl@ceph.osd.2.smithi002.stdout:Sep 17 00:26:26 smithi002 bash[76966]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-17T00:26:27.230 INFO:journalctl@ceph.osd.2.smithi002.stdout:Sep 17 00:26:26 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-2-activate[76984]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-17T00:26:27.230 INFO:journalctl@ceph.osd.2.smithi002.stdout:Sep 17 00:26:26 smithi002 bash[76966]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-17T00:26:27.695 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:26:27 smithi002 ceph-mon[66732]: from='client.44434 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:26:27.695 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:26:27 smithi002 ceph-mon[66732]: pgmap v106: 161 pgs: 12 stale+active+clean, 149 active+clean; 585 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-17T00:26:27.695 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:26:27 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:26:27.695 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:26:27 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T00:26:27.696 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:26:27 smithi002 ceph-mon[63804]: from='client.44434 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:26:27.696 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:26:27 smithi002 ceph-mon[63804]: pgmap v106: 161 pgs: 12 stale+active+clean, 149 active+clean; 585 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-17T00:26:27.696 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:26:27 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:26:27.696 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:26:27 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T00:26:27.716 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:26:27 smithi099 ceph-mon[56712]: from='client.44434 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:26:27.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:26:27 smithi099 ceph-mon[56712]: pgmap v106: 161 pgs: 12 stale+active+clean, 149 active+clean; 585 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-17T00:26:27.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:26:27 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:26:27.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:26:27 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T00:26:27.979 INFO:journalctl@ceph.osd.2.smithi002.stdout:Sep 17 00:26:27 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-2-activate[76984]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-17T00:26:27.979 INFO:journalctl@ceph.osd.2.smithi002.stdout:Sep 17 00:26:27 smithi002 bash[76966]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-17T00:26:27.979 INFO:journalctl@ceph.osd.2.smithi002.stdout:Sep 17 00:26:27 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-2-activate[76984]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-09-17T00:26:27.979 INFO:journalctl@ceph.osd.2.smithi002.stdout:Sep 17 00:26:27 smithi002 bash[76966]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-09-17T00:26:28.347 INFO:journalctl@ceph.osd.2.smithi002.stdout:Sep 17 00:26:27 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-2-activate[76984]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-09-17T00:26:28.347 INFO:journalctl@ceph.osd.2.smithi002.stdout:Sep 17 00:26:27 smithi002 bash[76966]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-09-17T00:26:28.347 INFO:journalctl@ceph.osd.2.smithi002.stdout:Sep 17 00:26:27 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-2-activate[76984]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-09-17T00:26:28.348 INFO:journalctl@ceph.osd.2.smithi002.stdout:Sep 17 00:26:27 smithi002 bash[76966]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-09-17T00:26:28.348 INFO:journalctl@ceph.osd.2.smithi002.stdout:Sep 17 00:26:27 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-2-activate[76984]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-09-17T00:26:28.348 INFO:journalctl@ceph.osd.2.smithi002.stdout:Sep 17 00:26:27 smithi002 bash[76966]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-09-17T00:26:28.348 INFO:journalctl@ceph.osd.2.smithi002.stdout:Sep 17 00:26:27 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-2-activate[76984]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-17T00:26:28.348 INFO:journalctl@ceph.osd.2.smithi002.stdout:Sep 17 00:26:27 smithi002 bash[76966]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-17T00:26:28.348 INFO:journalctl@ceph.osd.2.smithi002.stdout:Sep 17 00:26:27 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-2-activate[76984]: --> ceph-volume raw activate successful for osd ID: 2 2024-09-17T00:26:28.348 INFO:journalctl@ceph.osd.2.smithi002.stdout:Sep 17 00:26:27 smithi002 bash[76966]: --> ceph-volume raw activate successful for osd ID: 2 2024-09-17T00:26:28.348 INFO:journalctl@ceph.osd.2.smithi002.stdout:Sep 17 00:26:28 smithi002 podman[76966]: 2024-09-17 00:26:28.017205524 +0000 UTC m=+1.697946012 container died efc318ad5f0f352878ce8c512f1077841379192582668039451c5831b224c4a2 (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-2-activate, GIT_CLEAN=True, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-c2ddf97, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, ceph=True) 2024-09-17T00:26:28.713 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:26:28 smithi002 ceph-mon[63804]: Health check failed: Degraded data redundancy: 67/726 objects degraded (9.229%), 15 pgs degraded (PG_DEGRADED) 2024-09-17T00:26:28.714 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:26:28 smithi002 ceph-mon[66732]: Health check failed: Degraded data redundancy: 67/726 objects degraded (9.229%), 15 pgs degraded (PG_DEGRADED) 2024-09-17T00:26:28.716 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:26:28 smithi099 ceph-mon[56712]: Health check failed: Degraded data redundancy: 67/726 objects degraded (9.229%), 15 pgs degraded (PG_DEGRADED) 2024-09-17T00:26:28.717 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:26:28 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[52757]: ts=2024-09-17T00:26:28.660Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd name=CephOSDFlapping index=13 msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.1.1-330-gc2ddf972 (c2ddf9721ef254645201cdbabd9c8f49c30a586f) squid (stable)\", cluster=\"3aad27a2-7489-11ef-bceb-c7b262605968\", cluster_addr=\"172.21.15.2\", device_class=\"ssd\", hostname=\"smithi002\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.2\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.2\", device_class=\"ssd\", hostname=\"smithi002\", instance=\"172.21.15.99:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.2\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-17T00:26:28.979 INFO:journalctl@ceph.osd.2.smithi002.stdout:Sep 17 00:26:28 smithi002 podman[77116]: 2024-09-17 00:26:28.716821916 +0000 UTC m=+0.691391212 container remove efc318ad5f0f352878ce8c512f1077841379192582668039451c5831b224c4a2 (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-2-activate, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, RELEASE=squid-c2ddf97, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-17T00:26:28.979 INFO:journalctl@ceph.osd.2.smithi002.stdout:Sep 17 00:26:28 smithi002 podman[77176]: 2024-09-17 00:26:28.844328653 +0000 UTC m=+0.017649898 image pull ebb0615149def4a3179c82c0cc47af75808300abb9f1142788e5e605a7ac0e5b quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f 2024-09-17T00:26:28.979 INFO:journalctl@ceph.osd.2.smithi002.stdout:Sep 17 00:26:28 smithi002 podman[77176]: 2024-09-17 00:26:28.951048436 +0000 UTC m=+0.124369715 container create d21401b6763b6ee53aa5fd95e387f129c3cfec4c07c6de3fae4bda256a1e9d32 (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-2, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-c2ddf97, org.label-schema.schema-version=1.0) 2024-09-17T00:26:29.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:26:29 smithi099 ceph-mon[56712]: pgmap v107: 161 pgs: 32 active+undersized, 15 active+undersized+degraded, 114 active+clean; 585 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 67/726 objects degraded (9.229%) 2024-09-17T00:26:29.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:26:29 smithi002 ceph-mon[66732]: pgmap v107: 161 pgs: 32 active+undersized, 15 active+undersized+degraded, 114 active+clean; 585 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 67/726 objects degraded (9.229%) 2024-09-17T00:26:29.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:26:29 smithi002 ceph-mon[63804]: pgmap v107: 161 pgs: 32 active+undersized, 15 active+undersized+degraded, 114 active+clean; 585 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 67/726 objects degraded (9.229%) 2024-09-17T00:26:29.729 INFO:journalctl@ceph.osd.2.smithi002.stdout:Sep 17 00:26:29 smithi002 podman[77176]: 2024-09-17 00:26:29.370055937 +0000 UTC m=+0.543377178 container init d21401b6763b6ee53aa5fd95e387f129c3cfec4c07c6de3fae4bda256a1e9d32 (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-2, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-c2ddf97, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, io.buildah.version=1.37.2) 2024-09-17T00:26:29.729 INFO:journalctl@ceph.osd.2.smithi002.stdout:Sep 17 00:26:29 smithi002 podman[77176]: 2024-09-17 00:26:29.373869809 +0000 UTC m=+0.547191052 container start d21401b6763b6ee53aa5fd95e387f129c3cfec4c07c6de3fae4bda256a1e9d32 (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=squid-c2ddf97, org.label-schema.schema-version=1.0) 2024-09-17T00:26:29.730 INFO:journalctl@ceph.osd.2.smithi002.stdout:Sep 17 00:26:29 smithi002 ceph-osd[77194]: -- 172.21.15.2:0/4166556679 <== mon.0 v2:172.21.15.2:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5636d6adda40 con 0x5636d78a4400 2024-09-17T00:26:29.730 INFO:journalctl@ceph.osd.2.smithi002.stdout:Sep 17 00:26:29 smithi002 bash[77176]: d21401b6763b6ee53aa5fd95e387f129c3cfec4c07c6de3fae4bda256a1e9d32 2024-09-17T00:26:29.730 INFO:journalctl@ceph.osd.2.smithi002.stdout:Sep 17 00:26:29 smithi002 systemd[1]: Started Ceph osd.2 for 3aad27a2-7489-11ef-bceb-c7b262605968. 2024-09-17T00:26:30.785 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:26:30 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:26:30.785 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:26:30 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:26:30.785 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:26:30 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:26:30.785 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:26:30 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:26:30.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:26:30 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:26:30.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:26:30 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:26:31.098 INFO:journalctl@ceph.osd.2.smithi002.stdout:Sep 17 00:26:30 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-2[77190]: 2024-09-17T00:26:30.787+0000 7f3080537640 -1 Falling back to public interface 2024-09-17T00:26:31.826 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:26:31 smithi002 ceph-mon[66732]: pgmap v108: 161 pgs: 32 active+undersized, 15 active+undersized+degraded, 114 active+clean; 585 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 67/726 objects degraded (9.229%) 2024-09-17T00:26:31.826 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:26:31 smithi002 ceph-mon[63804]: pgmap v108: 161 pgs: 32 active+undersized, 15 active+undersized+degraded, 114 active+clean; 585 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 67/726 objects degraded (9.229%) 2024-09-17T00:26:31.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:26:31 smithi099 ceph-mon[56712]: pgmap v108: 161 pgs: 32 active+undersized, 15 active+undersized+degraded, 114 active+clean; 585 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 67/726 objects degraded (9.229%) 2024-09-17T00:26:33.625 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:26:33 smithi002 ceph-mon[66732]: pgmap v109: 161 pgs: 32 active+undersized, 15 active+undersized+degraded, 114 active+clean; 585 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 67/726 objects degraded (9.229%) 2024-09-17T00:26:33.626 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:26:33 smithi002 ceph-mon[63804]: pgmap v109: 161 pgs: 32 active+undersized, 15 active+undersized+degraded, 114 active+clean; 585 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 67/726 objects degraded (9.229%) 2024-09-17T00:26:33.969 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:26:33 smithi099 ceph-mon[56712]: pgmap v109: 161 pgs: 32 active+undersized, 15 active+undersized+degraded, 114 active+clean; 585 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 67/726 objects degraded (9.229%) 2024-09-17T00:26:33.969 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:26:33 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[52757]: ts=2024-09-17T00:26:33.506Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes name=CephNodeDiskspaceWarning index=4 msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi002\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi002\", job=\"node\", machine=\"x86_64\", nodename=\"smithi002\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"3aad27a2-7489-11ef-bceb-c7b262605968\", domainname=\"(none)\", instance=\"smithi002\", job=\"node\", machine=\"x86_64\", nodename=\"smithi002\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-17T00:26:34.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:26:34 smithi002 ceph-mon[66732]: from='osd.2 [v2:172.21.15.2:6818/1696463731,v1:172.21.15.2:6819/1696463731]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-17T00:26:34.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:26:34 smithi002 ceph-mon[66732]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-17T00:26:34.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:26:34 smithi002 ceph-mon[63804]: from='osd.2 [v2:172.21.15.2:6818/1696463731,v1:172.21.15.2:6819/1696463731]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-17T00:26:34.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:26:34 smithi002 ceph-mon[63804]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-17T00:26:34.730 INFO:journalctl@ceph.osd.2.smithi002.stdout:Sep 17 00:26:34 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-2[77190]: 2024-09-17T00:26:34.306+0000 7f3080537640 -1 osd.2 0 read_superblock omap replica is missing. 2024-09-17T00:26:34.730 INFO:journalctl@ceph.osd.2.smithi002.stdout:Sep 17 00:26:34 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-2[77190]: 2024-09-17T00:26:34.326+0000 7f3080537640 -1 osd.2 122 log_to_monitors true 2024-09-17T00:26:34.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:26:34 smithi099 ceph-mon[56712]: from='osd.2 [v2:172.21.15.2:6818/1696463731,v1:172.21.15.2:6819/1696463731]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-17T00:26:34.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:26:34 smithi099 ceph-mon[56712]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-17T00:26:35.229 INFO:journalctl@ceph.osd.2.smithi002.stdout:Sep 17 00:26:34 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-2[77190]: 2024-09-17T00:26:34.834+0000 7f30742da640 -1 osd.2 122 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T00:26:35.902 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:26:35 smithi002 ceph-mon[63804]: pgmap v110: 161 pgs: 32 active+undersized, 15 active+undersized+degraded, 114 active+clean; 585 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 67/726 objects degraded (9.229%) 2024-09-17T00:26:35.902 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:26:35 smithi002 ceph-mon[63804]: from='client.44398 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:26:35.902 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:26:35 smithi002 ceph-mon[63804]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-17T00:26:35.902 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:26:35 smithi002 ceph-mon[63804]: osdmap e125: 8 total, 7 up, 8 in 2024-09-17T00:26:35.902 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:26:35 smithi002 ceph-mon[63804]: from='osd.2 [v2:172.21.15.2:6818/1696463731,v1:172.21.15.2:6819/1696463731]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]: dispatch 2024-09-17T00:26:35.902 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:26:35 smithi002 ceph-mon[63804]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]: dispatch 2024-09-17T00:26:35.902 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:26:35 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:26:35.903 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:26:35 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:26:35.903 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:26:35 smithi002 ceph-mon[63804]: from='osd.2 ' entity='osd.2' 2024-09-17T00:26:35.903 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:26:35 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:26:35.903 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:26:35 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:26:35.903 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:26:35 smithi002 ceph-mon[66732]: pgmap v110: 161 pgs: 32 active+undersized, 15 active+undersized+degraded, 114 active+clean; 585 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 67/726 objects degraded (9.229%) 2024-09-17T00:26:35.903 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:26:35 smithi002 ceph-mon[66732]: from='client.44398 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:26:35.903 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:26:35 smithi002 ceph-mon[66732]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-17T00:26:35.903 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:26:35 smithi002 ceph-mon[66732]: osdmap e125: 8 total, 7 up, 8 in 2024-09-17T00:26:35.903 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:26:35 smithi002 ceph-mon[66732]: from='osd.2 [v2:172.21.15.2:6818/1696463731,v1:172.21.15.2:6819/1696463731]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]: dispatch 2024-09-17T00:26:35.903 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:26:35 smithi002 ceph-mon[66732]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]: dispatch 2024-09-17T00:26:35.903 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:26:35 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:26:35.904 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:26:35 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:26:35.904 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:26:35 smithi002 ceph-mon[66732]: from='osd.2 ' entity='osd.2' 2024-09-17T00:26:35.904 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:26:35 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:26:35.904 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:26:35 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:26:35.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:26:35 smithi099 ceph-mon[56712]: pgmap v110: 161 pgs: 32 active+undersized, 15 active+undersized+degraded, 114 active+clean; 585 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 67/726 objects degraded (9.229%) 2024-09-17T00:26:35.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:26:35 smithi099 ceph-mon[56712]: from='client.44398 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:26:35.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:26:35 smithi099 ceph-mon[56712]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-17T00:26:35.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:26:35 smithi099 ceph-mon[56712]: osdmap e125: 8 total, 7 up, 8 in 2024-09-17T00:26:35.968 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:26:35 smithi099 ceph-mon[56712]: from='osd.2 [v2:172.21.15.2:6818/1696463731,v1:172.21.15.2:6819/1696463731]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]: dispatch 2024-09-17T00:26:35.968 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:26:35 smithi099 ceph-mon[56712]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]: dispatch 2024-09-17T00:26:35.968 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:26:35 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:26:35.968 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:26:35 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:26:35.968 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:26:35 smithi099 ceph-mon[56712]: from='osd.2 ' entity='osd.2' 2024-09-17T00:26:35.968 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:26:35 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:26:35.968 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:26:35 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:26:36.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:26:36 smithi099 ceph-mon[56712]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-17T00:26:36.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:26:36 smithi099 ceph-mon[56712]: osd.2 [v2:172.21.15.2:6818/1696463731,v1:172.21.15.2:6819/1696463731] boot 2024-09-17T00:26:36.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:26:36 smithi099 ceph-mon[56712]: osdmap e126: 8 total, 8 up, 8 in 2024-09-17T00:26:36.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:26:36 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T00:26:36.968 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:26:36 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: ::ffff:172.21.15.99 - - [17/Sep/2024:00:26:36] "GET /metrics HTTP/1.1" 200 38425 "" "Prometheus/2.43.0" 2024-09-17T00:26:36.968 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:26:36 smithi002 ceph-mon[63804]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-17T00:26:36.968 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:26:36 smithi002 ceph-mon[63804]: osd.2 [v2:172.21.15.2:6818/1696463731,v1:172.21.15.2:6819/1696463731] boot 2024-09-17T00:26:36.968 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:26:36 smithi002 ceph-mon[63804]: osdmap e126: 8 total, 8 up, 8 in 2024-09-17T00:26:36.968 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:26:36 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T00:26:36.968 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:26:36 smithi002 ceph-mon[66732]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-17T00:26:36.968 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:26:36 smithi002 ceph-mon[66732]: osd.2 [v2:172.21.15.2:6818/1696463731,v1:172.21.15.2:6819/1696463731] boot 2024-09-17T00:26:36.968 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:26:36 smithi002 ceph-mon[66732]: osdmap e126: 8 total, 8 up, 8 in 2024-09-17T00:26:36.969 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:26:36 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-17T00:26:37.921 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:26:37 smithi002 ceph-mon[66732]: pgmap v113: 161 pgs: 32 active+undersized, 15 active+undersized+degraded, 114 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 67/726 objects degraded (9.229%) 2024-09-17T00:26:37.922 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:26:37 smithi002 ceph-mon[66732]: osdmap e127: 8 total, 8 up, 8 in 2024-09-17T00:26:37.922 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:26:37 smithi002 ceph-mon[63804]: pgmap v113: 161 pgs: 32 active+undersized, 15 active+undersized+degraded, 114 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 67/726 objects degraded (9.229%) 2024-09-17T00:26:37.922 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:26:37 smithi002 ceph-mon[63804]: osdmap e127: 8 total, 8 up, 8 in 2024-09-17T00:26:37.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:26:37 smithi099 ceph-mon[56712]: pgmap v113: 161 pgs: 32 active+undersized, 15 active+undersized+degraded, 114 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 67/726 objects degraded (9.229%) 2024-09-17T00:26:37.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:26:37 smithi099 ceph-mon[56712]: osdmap e127: 8 total, 8 up, 8 in 2024-09-17T00:26:38.824 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:26:38 smithi002 ceph-mon[66732]: pgmap v115: 161 pgs: 10 peering, 9 active+undersized, 3 active+undersized+degraded, 139 active+clean; 585 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 4/726 objects degraded (0.551%) 2024-09-17T00:26:38.824 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:26:38 smithi002 ceph-mon[66732]: Health check update: Degraded data redundancy: 4/726 objects degraded (0.551%), 3 pgs degraded (PG_DEGRADED) 2024-09-17T00:26:38.824 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:26:38 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:26:38.824 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:26:38 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:26:38.824 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:26:38 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:26:38.824 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:26:38 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:26:38.824 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:26:38 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:26:38.824 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:26:38 smithi002 ceph-mon[63804]: pgmap v115: 161 pgs: 10 peering, 9 active+undersized, 3 active+undersized+degraded, 139 active+clean; 585 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 4/726 objects degraded (0.551%) 2024-09-17T00:26:38.825 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:26:38 smithi002 ceph-mon[63804]: Health check update: Degraded data redundancy: 4/726 objects degraded (0.551%), 3 pgs degraded (PG_DEGRADED) 2024-09-17T00:26:38.825 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:26:38 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:26:38.825 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:26:38 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:26:38.825 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:26:38 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:26:38.825 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:26:38 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:26:38.825 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:26:38 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:26:38.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:26:38 smithi099 ceph-mon[56712]: pgmap v115: 161 pgs: 10 peering, 9 active+undersized, 3 active+undersized+degraded, 139 active+clean; 585 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 4/726 objects degraded (0.551%) 2024-09-17T00:26:38.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:26:38 smithi099 ceph-mon[56712]: Health check update: Degraded data redundancy: 4/726 objects degraded (0.551%), 3 pgs degraded (PG_DEGRADED) 2024-09-17T00:26:38.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:26:38 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:26:38.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:26:38 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:26:38.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:26:38 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:26:38.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:26:38 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:26:38.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:26:38 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:26:38.968 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:26:38 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[52757]: ts=2024-09-17T00:26:38.658Z caller=alerting.go:366 level=warn component="rule manager" alert="unsupported value type" msg="Expanding alert template failed" err="error executing template __alert_CephOSDDown: template: __alert_CephOSDDown:1:358: executing \"__alert_CephOSDDown\" at : error calling query: found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.1.1-330-gc2ddf972 (c2ddf9721ef254645201cdbabd9c8f49c30a586f) squid (stable)\", cluster=\"3aad27a2-7489-11ef-bceb-c7b262605968\", cluster_addr=\"172.21.15.2\", device_class=\"ssd\", hostname=\"smithi002\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.2\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.2\", device_class=\"ssd\", hostname=\"smithi002\", instance=\"172.21.15.99:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.2\"}];many-to-many matching not allowed: matching labels must be unique on one side" data="unsupported value type" 2024-09-17T00:26:38.968 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:26:38 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[52757]: ts=2024-09-17T00:26:38.660Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd name=CephOSDFlapping index=13 msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.1.1-330-gc2ddf972 (c2ddf9721ef254645201cdbabd9c8f49c30a586f) squid (stable)\", cluster=\"3aad27a2-7489-11ef-bceb-c7b262605968\", cluster_addr=\"172.21.15.2\", device_class=\"ssd\", hostname=\"smithi002\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.2\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.2\", device_class=\"ssd\", hostname=\"smithi002\", instance=\"172.21.15.99:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.2\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-17T00:26:39.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:26:39 smithi099 ceph-mon[56712]: Reconfiguring iscsi.foo.smithi002.vmouem (dependencies changed)... 2024-09-17T00:26:39.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:26:39 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi002.vmouem", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-17T00:26:39.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:26:39 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi002.vmouem", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-17T00:26:39.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:26:39 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:26:39.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:26:39 smithi099 ceph-mon[56712]: Reconfiguring daemon iscsi.foo.smithi002.vmouem on smithi002 2024-09-17T00:26:39.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:26:39 smithi002 ceph-mon[63804]: Reconfiguring iscsi.foo.smithi002.vmouem (dependencies changed)... 2024-09-17T00:26:39.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:26:39 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi002.vmouem", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-17T00:26:39.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:26:39 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi002.vmouem", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-17T00:26:39.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:26:39 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:26:39.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:26:39 smithi002 ceph-mon[63804]: Reconfiguring daemon iscsi.foo.smithi002.vmouem on smithi002 2024-09-17T00:26:39.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:26:39 smithi002 ceph-mon[66732]: Reconfiguring iscsi.foo.smithi002.vmouem (dependencies changed)... 2024-09-17T00:26:39.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:26:39 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi002.vmouem", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-17T00:26:39.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:26:39 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi002.vmouem", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-17T00:26:39.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:26:39 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:26:39.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:26:39 smithi002 ceph-mon[66732]: Reconfiguring daemon iscsi.foo.smithi002.vmouem on smithi002 2024-09-17T00:26:40.915 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:26:40 smithi002 ceph-mon[66732]: pgmap v116: 161 pgs: 10 peering, 151 active+clean; 585 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:26:40.916 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:26:40 smithi002 ceph-mon[63804]: pgmap v116: 161 pgs: 10 peering, 151 active+clean; 585 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:26:40.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:26:40 smithi099 ceph-mon[56712]: pgmap v116: 161 pgs: 10 peering, 151 active+clean; 585 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:26:41.608 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:26:41 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: 2024-09-17T00:26:41.490+0000 7f17e1e2b640 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi002.vmouem ... 2024-09-17T00:26:41.608 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:26:41 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: Creating ceph-iscsi config... 2024-09-17T00:26:41.608 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:26:41 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/iscsi-gateway.cfg 2024-09-17T00:26:41.608 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:26:41 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/tcmu-runner-entrypoint.sh 2024-09-17T00:26:41.609 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:26:41 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: Non-zero exit code 1 from systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem 2024-09-17T00:26:41.609 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:26:41 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: systemctl: stderr Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:26:41.609 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:26:41 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: systemctl: stderr See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:26:41.609 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:26:41 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: Traceback (most recent call last): 2024-09-17T00:26:41.609 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:26:41 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-17T00:26:41.609 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:26:41 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: return _run_code(code, main_globals, None, 2024-09-17T00:26:41.609 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:26:41 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-17T00:26:41.609 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:26:41 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: exec(code, run_globals) 2024-09-17T00:26:41.609 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:26:41 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5579, in 2024-09-17T00:26:41.609 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:26:41 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5567, in main 2024-09-17T00:26:41.609 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:26:41 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3049, in command_deploy_from 2024-09-17T00:26:41.610 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:26:41 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3084, in _common_deploy 2024-09-17T00:26:41.610 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:26:41 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3104, in _deploy_daemon_container 2024-09-17T00:26:41.610 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:26:41 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 1124, in deploy_daemon 2024-09-17T00:26:41.610 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:26:41 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/tmp/tmpnroy8d0w.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-17T00:26:41.610 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:26:41 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: RuntimeError: Failed command: systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem: Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:26:41.610 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:26:41 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:26:41.611 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:26:41 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: Traceback (most recent call last): 2024-09-17T00:26:41.611 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:26:41 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1134, in _check_daemons 2024-09-17T00:26:41.611 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:26:41 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: self.mgr._daemon_action(daemon_spec, action=action) 2024-09-17T00:26:41.611 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:26:41 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2473, in _daemon_action 2024-09-17T00:26:41.611 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:26:41 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: return self.wait_async( 2024-09-17T00:26:41.611 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:26:41 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/usr/share/ceph/mgr/cephadm/module.py", line 796, in wait_async 2024-09-17T00:26:41.611 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:26:41 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: return self.event_loop.get_result(coro, timeout) 2024-09-17T00:26:41.611 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:26:41 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-09-17T00:26:41.611 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:26:41 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: return future.result(timeout) 2024-09-17T00:26:41.611 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:26:41 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/lib64/python3.9/concurrent/futures/_base.py", line 446, in result 2024-09-17T00:26:41.612 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:26:41 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: return self.__get_result() 2024-09-17T00:26:41.612 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:26:41 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/lib64/python3.9/concurrent/futures/_base.py", line 391, in __get_result 2024-09-17T00:26:41.612 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:26:41 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: raise self._exception 2024-09-17T00:26:41.612 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:26:41 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1370, in _create_daemon 2024-09-17T00:26:41.612 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:26:41 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: out, err, code = await self._run_cephadm( 2024-09-17T00:26:41.612 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:26:41 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1713, in _run_cephadm 2024-09-17T00:26:41.612 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:26:41 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: raise OrchestratorError( 2024-09-17T00:26:41.612 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:26:41 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi002.vmouem ... 2024-09-17T00:26:41.612 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:26:41 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: Creating ceph-iscsi config... 2024-09-17T00:26:41.612 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:26:41 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/iscsi-gateway.cfg 2024-09-17T00:26:41.612 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:26:41 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/tcmu-runner-entrypoint.sh 2024-09-17T00:26:41.613 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:26:41 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: Non-zero exit code 1 from systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem 2024-09-17T00:26:41.613 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:26:41 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: systemctl: stderr Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:26:41.613 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:26:41 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: systemctl: stderr See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:26:41.613 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:26:41 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: Traceback (most recent call last): 2024-09-17T00:26:41.613 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:26:41 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-17T00:26:41.613 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:26:41 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: return _run_code(code, main_globals, None, 2024-09-17T00:26:41.613 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:26:41 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-17T00:26:41.613 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:26:41 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: exec(code, run_globals) 2024-09-17T00:26:41.613 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:26:41 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5579, in 2024-09-17T00:26:41.613 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:26:41 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5567, in main 2024-09-17T00:26:41.613 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:26:41 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3049, in command_deploy_from 2024-09-17T00:26:41.614 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:26:41 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3084, in _common_deploy 2024-09-17T00:26:41.614 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:26:41 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3104, in _deploy_daemon_container 2024-09-17T00:26:41.614 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:26:41 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 1124, in deploy_daemon 2024-09-17T00:26:41.614 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:26:41 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/tmp/tmpnroy8d0w.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-17T00:26:41.614 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:26:41 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: RuntimeError: Failed command: systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem: Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:26:41.614 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:26:41 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:26:41.614 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:26:41 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: 2024-09-17T00:26:41.534+0000 7f17efec7640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (1 PGs are or would become offline) 2024-09-17T00:26:41.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:26:41 smithi099 ceph-mon[56712]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 4/726 objects degraded (0.551%), 3 pgs degraded) 2024-09-17T00:26:41.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:26:41 smithi099 ceph-mon[56712]: Cluster is now healthy 2024-09-17T00:26:41.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:26:41 smithi099 ceph-mon[56712]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi002.vmouem ... 2024-09-17T00:26:41.967 INFO:journalctl@ceph.mon.b.smithi099.stdout: Creating ceph-iscsi config... 2024-09-17T00:26:41.968 INFO:journalctl@ceph.mon.b.smithi099.stdout: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/iscsi-gateway.cfg 2024-09-17T00:26:41.968 INFO:journalctl@ceph.mon.b.smithi099.stdout: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/tcmu-runner-entrypoint.sh 2024-09-17T00:26:41.968 INFO:journalctl@ceph.mon.b.smithi099.stdout: Non-zero exit code 1 from systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem 2024-09-17T00:26:41.968 INFO:journalctl@ceph.mon.b.smithi099.stdout: systemctl: stderr Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:26:41.968 INFO:journalctl@ceph.mon.b.smithi099.stdout: systemctl: stderr See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:26:41.968 INFO:journalctl@ceph.mon.b.smithi099.stdout: Traceback (most recent call last): 2024-09-17T00:26:41.968 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-17T00:26:41.968 INFO:journalctl@ceph.mon.b.smithi099.stdout: return _run_code(code, main_globals, None, 2024-09-17T00:26:41.968 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-17T00:26:41.968 INFO:journalctl@ceph.mon.b.smithi099.stdout: exec(code, run_globals) 2024-09-17T00:26:41.968 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5579, in 2024-09-17T00:26:41.968 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5567, in main 2024-09-17T00:26:41.968 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3049, in command_deploy_from 2024-09-17T00:26:41.968 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3084, in _common_deploy 2024-09-17T00:26:41.969 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3104, in _deploy_daemon_container 2024-09-17T00:26:41.969 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 1124, in deploy_daemon 2024-09-17T00:26:41.969 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-17T00:26:41.969 INFO:journalctl@ceph.mon.b.smithi099.stdout: RuntimeError: Failed command: systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem: Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:26:41.969 INFO:journalctl@ceph.mon.b.smithi099.stdout: See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:26:41.969 INFO:journalctl@ceph.mon.b.smithi099.stdout: Traceback (most recent call last): 2024-09-17T00:26:41.969 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1134, in _check_daemons 2024-09-17T00:26:41.969 INFO:journalctl@ceph.mon.b.smithi099.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-09-17T00:26:41.969 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2473, in _daemon_action 2024-09-17T00:26:41.969 INFO:journalctl@ceph.mon.b.smithi099.stdout: return self.wait_async( 2024-09-17T00:26:41.969 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 796, in wait_async 2024-09-17T00:26:41.969 INFO:journalctl@ceph.mon.b.smithi099.stdout: return self.event_loop.get_result(coro, timeout) 2024-09-17T00:26:41.969 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-09-17T00:26:41.969 INFO:journalctl@ceph.mon.b.smithi099.stdout: return future.result(timeout) 2024-09-17T00:26:41.970 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/lib64/python3.9/concurrent/futures/_base.py", line 446, in result 2024-09-17T00:26:41.970 INFO:journalctl@ceph.mon.b.smithi099.stdout: return self.__get_result() 2024-09-17T00:26:41.970 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/lib64/python3.9/concurrent/futures/_base.py", line 391, in __get_result 2024-09-17T00:26:41.970 INFO:journalctl@ceph.mon.b.smithi099.stdout: raise self._exception 2024-09-17T00:26:41.970 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1370, in _create_daemon 2024-09-17T00:26:41.970 INFO:journalctl@ceph.mon.b.smithi099.stdout: out, err, code = await self._run_cephadm( 2024-09-17T00:26:41.970 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1713, in _run_cephadm 2024-09-17T00:26:41.970 INFO:journalctl@ceph.mon.b.smithi099.stdout: raise OrchestratorError( 2024-09-17T00:26:41.970 INFO:journalctl@ceph.mon.b.smithi099.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi002.vmouem ... 2024-09-17T00:26:41.970 INFO:journalctl@ceph.mon.b.smithi099.stdout: Creating ceph-iscsi config... 2024-09-17T00:26:41.970 INFO:journalctl@ceph.mon.b.smithi099.stdout: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/iscsi-gateway.cfg 2024-09-17T00:26:41.970 INFO:journalctl@ceph.mon.b.smithi099.stdout: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/tcmu-runner-entrypoint.sh 2024-09-17T00:26:41.970 INFO:journalctl@ceph.mon.b.smithi099.stdout: Non-zero exit code 1 from systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem 2024-09-17T00:26:41.970 INFO:journalctl@ceph.mon.b.smithi099.stdout: systemctl: stderr Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:26:41.971 INFO:journalctl@ceph.mon.b.smithi099.stdout: systemctl: stderr See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:26:41.971 INFO:journalctl@ceph.mon.b.smithi099.stdout: Traceback (most recent call last): 2024-09-17T00:26:41.971 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-17T00:26:41.971 INFO:journalctl@ceph.mon.b.smithi099.stdout: return _run_code(code, main_globals, None, 2024-09-17T00:26:41.971 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-17T00:26:41.971 INFO:journalctl@ceph.mon.b.smithi099.stdout: exec(code, run_globals) 2024-09-17T00:26:41.971 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5579, in 2024-09-17T00:26:41.971 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5567, in main 2024-09-17T00:26:41.971 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3049, in command_deploy_from 2024-09-17T00:26:41.971 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3084, in _common_deploy 2024-09-17T00:26:41.971 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3104, in _deploy_daemon_container 2024-09-17T00:26:41.971 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 1124, in deploy_daemon 2024-09-17T00:26:41.971 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-17T00:26:41.971 INFO:journalctl@ceph.mon.b.smithi099.stdout: RuntimeError: Failed command: systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem: Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:26:41.972 INFO:journalctl@ceph.mon.b.smithi099.stdout: See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:26:41.972 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:26:41 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T00:26:41.972 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:26:41 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:26:41.972 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:26:41 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:26:41.972 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:26:41 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:26:41.972 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:26:41 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-17T00:26:41.972 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:26:41 smithi099 ceph-mon[56712]: from='mon.1 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-17T00:26:41.972 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:26:41 smithi099 ceph-mon[56712]: Upgrade: unsafe to stop osd(s) at this time (1 PGs are or would become offline) 2024-09-17T00:26:41.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:26:41 smithi002 ceph-mon[66732]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 4/726 objects degraded (0.551%), 3 pgs degraded) 2024-09-17T00:26:41.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:26:41 smithi002 ceph-mon[66732]: Cluster is now healthy 2024-09-17T00:26:41.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:26:41 smithi002 ceph-mon[66732]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi002.vmouem ... 2024-09-17T00:26:41.979 INFO:journalctl@ceph.mon.c.smithi002.stdout: Creating ceph-iscsi config... 2024-09-17T00:26:41.979 INFO:journalctl@ceph.mon.c.smithi002.stdout: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/iscsi-gateway.cfg 2024-09-17T00:26:41.979 INFO:journalctl@ceph.mon.c.smithi002.stdout: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/tcmu-runner-entrypoint.sh 2024-09-17T00:26:41.980 INFO:journalctl@ceph.mon.c.smithi002.stdout: Non-zero exit code 1 from systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem 2024-09-17T00:26:41.980 INFO:journalctl@ceph.mon.c.smithi002.stdout: systemctl: stderr Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:26:41.980 INFO:journalctl@ceph.mon.c.smithi002.stdout: systemctl: stderr See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:26:41.980 INFO:journalctl@ceph.mon.c.smithi002.stdout: Traceback (most recent call last): 2024-09-17T00:26:41.980 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-17T00:26:41.980 INFO:journalctl@ceph.mon.c.smithi002.stdout: return _run_code(code, main_globals, None, 2024-09-17T00:26:41.980 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-17T00:26:41.980 INFO:journalctl@ceph.mon.c.smithi002.stdout: exec(code, run_globals) 2024-09-17T00:26:41.980 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5579, in 2024-09-17T00:26:41.980 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5567, in main 2024-09-17T00:26:41.980 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3049, in command_deploy_from 2024-09-17T00:26:41.980 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3084, in _common_deploy 2024-09-17T00:26:41.981 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3104, in _deploy_daemon_container 2024-09-17T00:26:41.981 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 1124, in deploy_daemon 2024-09-17T00:26:41.981 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-17T00:26:41.981 INFO:journalctl@ceph.mon.c.smithi002.stdout: RuntimeError: Failed command: systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem: Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:26:41.981 INFO:journalctl@ceph.mon.c.smithi002.stdout: See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:26:41.981 INFO:journalctl@ceph.mon.c.smithi002.stdout: Traceback (most recent call last): 2024-09-17T00:26:41.981 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1134, in _check_daemons 2024-09-17T00:26:41.981 INFO:journalctl@ceph.mon.c.smithi002.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-09-17T00:26:41.981 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2473, in _daemon_action 2024-09-17T00:26:41.981 INFO:journalctl@ceph.mon.c.smithi002.stdout: return self.wait_async( 2024-09-17T00:26:41.981 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 796, in wait_async 2024-09-17T00:26:41.982 INFO:journalctl@ceph.mon.c.smithi002.stdout: return self.event_loop.get_result(coro, timeout) 2024-09-17T00:26:41.982 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:26:41 smithi002 ceph-mon[63804]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 4/726 objects degraded (0.551%), 3 pgs degraded) 2024-09-17T00:26:41.982 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:26:41 smithi002 ceph-mon[63804]: Cluster is now healthy 2024-09-17T00:26:41.982 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:26:41 smithi002 ceph-mon[63804]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi002.vmouem ... 2024-09-17T00:26:41.982 INFO:journalctl@ceph.mon.a.smithi002.stdout: Creating ceph-iscsi config... 2024-09-17T00:26:41.982 INFO:journalctl@ceph.mon.a.smithi002.stdout: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/iscsi-gateway.cfg 2024-09-17T00:26:41.982 INFO:journalctl@ceph.mon.a.smithi002.stdout: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/tcmu-runner-entrypoint.sh 2024-09-17T00:26:41.982 INFO:journalctl@ceph.mon.a.smithi002.stdout: Non-zero exit code 1 from systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem 2024-09-17T00:26:41.982 INFO:journalctl@ceph.mon.a.smithi002.stdout: systemctl: stderr Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:26:41.983 INFO:journalctl@ceph.mon.a.smithi002.stdout: systemctl: stderr See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:26:41.983 INFO:journalctl@ceph.mon.a.smithi002.stdout: Traceback (most recent call last): 2024-09-17T00:26:41.983 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-17T00:26:41.983 INFO:journalctl@ceph.mon.a.smithi002.stdout: return _run_code(code, main_globals, None, 2024-09-17T00:26:41.983 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-17T00:26:41.983 INFO:journalctl@ceph.mon.a.smithi002.stdout: exec(code, run_globals) 2024-09-17T00:26:41.983 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5579, in 2024-09-17T00:26:41.983 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5567, in main 2024-09-17T00:26:41.983 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3049, in command_deploy_from 2024-09-17T00:26:41.983 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3084, in _common_deploy 2024-09-17T00:26:41.983 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3104, in _deploy_daemon_container 2024-09-17T00:26:41.984 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 1124, in deploy_daemon 2024-09-17T00:26:41.984 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-17T00:26:41.984 INFO:journalctl@ceph.mon.a.smithi002.stdout: RuntimeError: Failed command: systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem: Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:26:41.984 INFO:journalctl@ceph.mon.a.smithi002.stdout: See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:26:41.984 INFO:journalctl@ceph.mon.a.smithi002.stdout: Traceback (most recent call last): 2024-09-17T00:26:41.984 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1134, in _check_daemons 2024-09-17T00:26:41.984 INFO:journalctl@ceph.mon.a.smithi002.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-09-17T00:26:41.984 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2473, in _daemon_action 2024-09-17T00:26:41.984 INFO:journalctl@ceph.mon.a.smithi002.stdout: return self.wait_async( 2024-09-17T00:26:41.984 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 796, in wait_async 2024-09-17T00:26:41.984 INFO:journalctl@ceph.mon.a.smithi002.stdout: return self.event_loop.get_result(coro, timeout) 2024-09-17T00:26:41.985 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-09-17T00:26:41.985 INFO:journalctl@ceph.mon.c.smithi002.stdout: return future.result(timeout) 2024-09-17T00:26:41.985 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/lib64/python3.9/concurrent/futures/_base.py", line 446, in result 2024-09-17T00:26:41.985 INFO:journalctl@ceph.mon.c.smithi002.stdout: return self.__get_result() 2024-09-17T00:26:41.985 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/lib64/python3.9/concurrent/futures/_base.py", line 391, in __get_result 2024-09-17T00:26:41.985 INFO:journalctl@ceph.mon.c.smithi002.stdout: raise self._exception 2024-09-17T00:26:41.985 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1370, in _create_daemon 2024-09-17T00:26:41.985 INFO:journalctl@ceph.mon.c.smithi002.stdout: out, err, code = await self._run_cephadm( 2024-09-17T00:26:41.986 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1713, in _run_cephadm 2024-09-17T00:26:41.986 INFO:journalctl@ceph.mon.c.smithi002.stdout: raise OrchestratorError( 2024-09-17T00:26:41.986 INFO:journalctl@ceph.mon.c.smithi002.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi002.vmouem ... 2024-09-17T00:26:41.986 INFO:journalctl@ceph.mon.c.smithi002.stdout: Creating ceph-iscsi config... 2024-09-17T00:26:41.986 INFO:journalctl@ceph.mon.c.smithi002.stdout: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/iscsi-gateway.cfg 2024-09-17T00:26:41.986 INFO:journalctl@ceph.mon.c.smithi002.stdout: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/tcmu-runner-entrypoint.sh 2024-09-17T00:26:41.986 INFO:journalctl@ceph.mon.c.smithi002.stdout: Non-zero exit code 1 from systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem 2024-09-17T00:26:41.986 INFO:journalctl@ceph.mon.c.smithi002.stdout: systemctl: stderr Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:26:41.986 INFO:journalctl@ceph.mon.c.smithi002.stdout: systemctl: stderr See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:26:41.986 INFO:journalctl@ceph.mon.c.smithi002.stdout: Traceback (most recent call last): 2024-09-17T00:26:41.986 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-17T00:26:41.986 INFO:journalctl@ceph.mon.c.smithi002.stdout: return _run_code(code, main_globals, None, 2024-09-17T00:26:41.987 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-17T00:26:41.987 INFO:journalctl@ceph.mon.c.smithi002.stdout: exec(code, run_globals) 2024-09-17T00:26:41.987 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5579, in 2024-09-17T00:26:41.987 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5567, in main 2024-09-17T00:26:41.987 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3049, in command_deploy_from 2024-09-17T00:26:41.987 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3084, in _common_deploy 2024-09-17T00:26:41.987 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3104, in _deploy_daemon_container 2024-09-17T00:26:41.987 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 1124, in deploy_daemon 2024-09-17T00:26:41.987 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-17T00:26:41.987 INFO:journalctl@ceph.mon.c.smithi002.stdout: RuntimeError: Failed command: systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem: Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:26:41.987 INFO:journalctl@ceph.mon.c.smithi002.stdout: See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:26:41.988 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-09-17T00:26:41.988 INFO:journalctl@ceph.mon.a.smithi002.stdout: return future.result(timeout) 2024-09-17T00:26:41.988 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/lib64/python3.9/concurrent/futures/_base.py", line 446, in result 2024-09-17T00:26:41.988 INFO:journalctl@ceph.mon.a.smithi002.stdout: return self.__get_result() 2024-09-17T00:26:41.988 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/lib64/python3.9/concurrent/futures/_base.py", line 391, in __get_result 2024-09-17T00:26:41.988 INFO:journalctl@ceph.mon.a.smithi002.stdout: raise self._exception 2024-09-17T00:26:41.988 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1370, in _create_daemon 2024-09-17T00:26:41.988 INFO:journalctl@ceph.mon.a.smithi002.stdout: out, err, code = await self._run_cephadm( 2024-09-17T00:26:41.989 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1713, in _run_cephadm 2024-09-17T00:26:41.989 INFO:journalctl@ceph.mon.a.smithi002.stdout: raise OrchestratorError( 2024-09-17T00:26:41.989 INFO:journalctl@ceph.mon.a.smithi002.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi002.vmouem ... 2024-09-17T00:26:41.989 INFO:journalctl@ceph.mon.a.smithi002.stdout: Creating ceph-iscsi config... 2024-09-17T00:26:41.989 INFO:journalctl@ceph.mon.a.smithi002.stdout: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/iscsi-gateway.cfg 2024-09-17T00:26:41.989 INFO:journalctl@ceph.mon.a.smithi002.stdout: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/tcmu-runner-entrypoint.sh 2024-09-17T00:26:41.989 INFO:journalctl@ceph.mon.a.smithi002.stdout: Non-zero exit code 1 from systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem 2024-09-17T00:26:41.989 INFO:journalctl@ceph.mon.a.smithi002.stdout: systemctl: stderr Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:26:41.989 INFO:journalctl@ceph.mon.a.smithi002.stdout: systemctl: stderr See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:26:41.989 INFO:journalctl@ceph.mon.a.smithi002.stdout: Traceback (most recent call last): 2024-09-17T00:26:41.989 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-17T00:26:41.989 INFO:journalctl@ceph.mon.a.smithi002.stdout: return _run_code(code, main_globals, None, 2024-09-17T00:26:41.990 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-17T00:26:41.990 INFO:journalctl@ceph.mon.a.smithi002.stdout: exec(code, run_globals) 2024-09-17T00:26:41.990 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5579, in 2024-09-17T00:26:41.990 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5567, in main 2024-09-17T00:26:41.990 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3049, in command_deploy_from 2024-09-17T00:26:41.990 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3084, in _common_deploy 2024-09-17T00:26:41.990 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3104, in _deploy_daemon_container 2024-09-17T00:26:41.990 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 1124, in deploy_daemon 2024-09-17T00:26:41.990 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-17T00:26:41.990 INFO:journalctl@ceph.mon.a.smithi002.stdout: RuntimeError: Failed command: systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem: Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:26:41.990 INFO:journalctl@ceph.mon.a.smithi002.stdout: See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:26:41.991 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:26:41 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T00:26:41.991 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:26:41 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:26:41.991 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:26:41 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:26:41.991 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:26:41 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:26:41.991 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:26:41 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-17T00:26:41.991 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:26:41 smithi002 ceph-mon[66732]: from='mon.1 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-17T00:26:41.991 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:26:41 smithi002 ceph-mon[66732]: Upgrade: unsafe to stop osd(s) at this time (1 PGs are or would become offline) 2024-09-17T00:26:41.991 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:26:41 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T00:26:41.992 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:26:41 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:26:41.992 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:26:41 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:26:41.992 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:26:41 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:26:41.992 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:26:41 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-17T00:26:41.992 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:26:41 smithi002 ceph-mon[63804]: from='mon.1 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-17T00:26:41.992 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:26:41 smithi002 ceph-mon[63804]: Upgrade: unsafe to stop osd(s) at this time (1 PGs are or would become offline) 2024-09-17T00:26:43.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:26:42 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:26:43.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:26:42 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T00:26:43.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:26:42 smithi099 ceph-mon[56712]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 672 B/s rd, 0 op/s 2024-09-17T00:26:43.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:26:42 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:26:43.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:26:42 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T00:26:43.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:26:42 smithi002 ceph-mon[63804]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 672 B/s rd, 0 op/s 2024-09-17T00:26:43.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:26:42 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:26:43.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:26:42 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T00:26:43.230 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:26:42 smithi002 ceph-mon[66732]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 672 B/s rd, 0 op/s 2024-09-17T00:26:43.967 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:26:43 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[52757]: ts=2024-09-17T00:26:43.506Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes name=CephNodeDiskspaceWarning index=4 msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi002\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi002\", job=\"node\", machine=\"x86_64\", nodename=\"smithi002\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"3aad27a2-7489-11ef-bceb-c7b262605968\", domainname=\"(none)\", instance=\"smithi002\", job=\"node\", machine=\"x86_64\", nodename=\"smithi002\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-17T00:26:45.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:26:45 smithi099 ceph-mon[56712]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 983 B/s rd, 1 op/s 2024-09-17T00:26:45.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:26:45 smithi002 ceph-mon[63804]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 983 B/s rd, 1 op/s 2024-09-17T00:26:45.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:26:45 smithi002 ceph-mon[66732]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 983 B/s rd, 1 op/s 2024-09-17T00:26:46.981 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:26:46 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: ::ffff:172.21.15.99 - - [17/Sep/2024:00:26:46] "GET /metrics HTTP/1.1" 200 38433 "" "Prometheus/2.43.0" 2024-09-17T00:26:47.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:26:47 smithi002 ceph-mon[66732]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 818 B/s rd, 0 op/s 2024-09-17T00:26:47.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:26:47 smithi002 ceph-mon[63804]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 818 B/s rd, 0 op/s 2024-09-17T00:26:47.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:26:47 smithi099 ceph-mon[56712]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 818 B/s rd, 0 op/s 2024-09-17T00:26:48.967 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:26:48 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[52757]: ts=2024-09-17T00:26:48.659Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd name=CephOSDFlapping index=13 msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.1.1-330-gc2ddf972 (c2ddf9721ef254645201cdbabd9c8f49c30a586f) squid (stable)\", cluster=\"3aad27a2-7489-11ef-bceb-c7b262605968\", cluster_addr=\"172.21.15.2\", device_class=\"ssd\", hostname=\"smithi002\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.2\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.2\", device_class=\"ssd\", hostname=\"smithi002\", instance=\"172.21.15.99:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.2\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-17T00:26:49.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:26:49 smithi002 ceph-mon[66732]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 723 B/s rd, 0 op/s 2024-09-17T00:26:49.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:26:49 smithi002 ceph-mon[63804]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 723 B/s rd, 0 op/s 2024-09-17T00:26:49.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:26:49 smithi099 ceph-mon[56712]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 723 B/s rd, 0 op/s 2024-09-17T00:26:51.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:26:51 smithi002 ceph-mon[66732]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-09-17T00:26:51.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:26:51 smithi002 ceph-mon[63804]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-09-17T00:26:51.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:26:51 smithi099 ceph-mon[56712]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-09-17T00:26:52.998 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:26:52 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:26:52.999 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:26:52 smithi002 ceph-mon[66732]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-09-17T00:26:52.999 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:26:52 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:26:52.999 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:26:52 smithi002 ceph-mon[63804]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-09-17T00:26:53.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:26:52 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:26:53.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:26:52 smithi099 ceph-mon[56712]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-09-17T00:26:53.817 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:26:53 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[52757]: ts=2024-09-17T00:26:53.505Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes name=CephNodeDiskspaceWarning index=4 msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi002\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi002\", job=\"node\", machine=\"x86_64\", nodename=\"smithi002\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"3aad27a2-7489-11ef-bceb-c7b262605968\", domainname=\"(none)\", instance=\"smithi002\", job=\"node\", machine=\"x86_64\", nodename=\"smithi002\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-17T00:26:54.216 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:26:53 smithi099 ceph-mon[56712]: from='client.? 172.21.15.2:0/3344884716' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-17T00:26:54.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:26:53 smithi002 ceph-mon[63804]: from='client.? 172.21.15.2:0/3344884716' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-17T00:26:54.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:26:53 smithi002 ceph-mon[66732]: from='client.? 172.21.15.2:0/3344884716' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-17T00:26:55.118 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:26:54 smithi099 ceph-mon[56712]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-09-17T00:26:55.313 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:26:54 smithi002 ceph-mon[63804]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-09-17T00:26:55.313 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:26:54 smithi002 ceph-mon[66732]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-09-17T00:26:56.705 INFO:teuthology.orchestra.run.smithi002.stdout:true 2024-09-17T00:26:56.939 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:26:56 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: ::ffff:172.21.15.99 - - [17/Sep/2024:00:26:56] "GET /metrics HTTP/1.1" 200 38433 "" "Prometheus/2.43.0" 2024-09-17T00:26:57.414 INFO:teuthology.orchestra.run.smithi002.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-17T00:26:57.414 INFO:teuthology.orchestra.run.smithi002.stdout:alertmanager.a smithi002 *:9093,9094 running (4m) 22s ago 9m 26.1M - 0.25.0 c8568f914cd2 a21e13b3d677 2024-09-17T00:26:57.414 INFO:teuthology.orchestra.run.smithi002.stdout:grafana.a smithi099 *:3000 running (4m) 2m ago 9m 46.9M - 8.3.5 dad864ee21e9 bafb824a3c08 2024-09-17T00:26:57.415 INFO:teuthology.orchestra.run.smithi002.stdout:iscsi.foo.smithi002.vmouem smithi002 running (43s) 22s ago 8m 48.3M - 3.5 e1d6a67b021e 7978a1bfe3f0 2024-09-17T00:26:57.415 INFO:teuthology.orchestra.run.smithi002.stdout:mgr.x smithi099 *:8443,9283,8765 running (3m) 2m ago 14m 439M - 19.1.1-330-gc2ddf972 ebb0615149de 6aa2ccef9ca0 2024-09-17T00:26:57.415 INFO:teuthology.orchestra.run.smithi002.stdout:mgr.y smithi002 *:8443,9283,8765 running (4m) 22s ago 15m 504M - 19.1.1-330-gc2ddf972 ebb0615149de 8838897e8449 2024-09-17T00:26:57.415 INFO:teuthology.orchestra.run.smithi002.stdout:mon.a smithi002 running (3m) 22s ago 16m 49.9M 2048M 19.1.1-330-gc2ddf972 ebb0615149de a035def53ef6 2024-09-17T00:26:57.415 INFO:teuthology.orchestra.run.smithi002.stdout:mon.b smithi099 running (2m) 2m ago 14m 17.5M 2048M 19.1.1-330-gc2ddf972 ebb0615149de aab3ab73a850 2024-09-17T00:26:57.415 INFO:teuthology.orchestra.run.smithi002.stdout:mon.c smithi002 running (2m) 22s ago 14m 42.0M 2048M 19.1.1-330-gc2ddf972 ebb0615149de 8a510255ae66 2024-09-17T00:26:57.415 INFO:teuthology.orchestra.run.smithi002.stdout:node-exporter.a smithi002 *:9100 running (4m) 22s ago 10m 15.4M - 1.5.0 0da6a335fe13 1030ae01bb32 2024-09-17T00:26:57.415 INFO:teuthology.orchestra.run.smithi002.stdout:node-exporter.b smithi099 *:9100 running (4m) 2m ago 10m 16.8M - 1.5.0 0da6a335fe13 01fcd35dca8a 2024-09-17T00:26:57.415 INFO:teuthology.orchestra.run.smithi002.stdout:osd.0 smithi002 running (106s) 22s ago 13m 44.4M 2737M 19.1.1-330-gc2ddf972 ebb0615149de 8345719cb6f0 2024-09-17T00:26:57.415 INFO:teuthology.orchestra.run.smithi002.stdout:osd.1 smithi002 running (67s) 22s ago 13m 51.0M 2737M 19.1.1-330-gc2ddf972 ebb0615149de 3dbc23d57c53 2024-09-17T00:26:57.415 INFO:teuthology.orchestra.run.smithi002.stdout:osd.2 smithi002 running (28s) 22s ago 13m 11.4M 2737M 19.1.1-330-gc2ddf972 ebb0615149de d21401b6763b 2024-09-17T00:26:57.415 INFO:teuthology.orchestra.run.smithi002.stdout:osd.3 smithi002 running (12m) 22s ago 12m 59.5M 2737M 17.2.0 e1d6a67b021e fd99d3764412 2024-09-17T00:26:57.415 INFO:teuthology.orchestra.run.smithi002.stdout:osd.4 smithi099 running (12m) 2m ago 12m 55.4M 2992M 17.2.0 e1d6a67b021e 0af1827b658d 2024-09-17T00:26:57.416 INFO:teuthology.orchestra.run.smithi002.stdout:osd.5 smithi099 running (12m) 2m ago 12m 54.2M 2992M 17.2.0 e1d6a67b021e 4f7ef3aefc58 2024-09-17T00:26:57.416 INFO:teuthology.orchestra.run.smithi002.stdout:osd.6 smithi099 running (11m) 2m ago 11m 51.0M 2992M 17.2.0 e1d6a67b021e 3213f5e6952a 2024-09-17T00:26:57.416 INFO:teuthology.orchestra.run.smithi002.stdout:osd.7 smithi099 running (11m) 2m ago 11m 56.6M 2992M 17.2.0 e1d6a67b021e 63be3a1ed0a1 2024-09-17T00:26:57.416 INFO:teuthology.orchestra.run.smithi002.stdout:prometheus.a smithi099 *:9095 running (3m) 2m ago 10m 50.1M - 2.43.0 a07b618ecd1d bf32c2ae11dc 2024-09-17T00:26:57.416 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.foo.smithi002.uwdhcs smithi002 *:8000 running (8m) 22s ago 8m 76.0M - 17.2.0 e1d6a67b021e 9c35813f459a 2024-09-17T00:26:57.416 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.foo.smithi099.utzfap smithi099 *:8000 running (8m) 2m ago 8m 76.0M - 17.2.0 e1d6a67b021e 34ec989160a6 2024-09-17T00:26:57.416 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.smpl.smithi002.rqnxii smithi002 *:80 running (8m) 22s ago 8m 74.2M - 17.2.0 e1d6a67b021e 0e180d0afb99 2024-09-17T00:26:57.416 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.smpl.smithi099.nzvkpm smithi099 *:80 running (8m) 2m ago 8m 76.5M - 17.2.0 e1d6a67b021e 6a068e82ac72 2024-09-17T00:26:57.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:26:57 smithi002 ceph-mon[66732]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 162 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:26:57.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:26:57 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-17T00:26:57.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:26:57 smithi002 ceph-mon[66732]: from='mon.1 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-17T00:26:57.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:26:57 smithi002 ceph-mon[66732]: Upgrade: osd.3 is safe to restart 2024-09-17T00:26:57.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:26:57 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T00:26:57.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:26:57 smithi002 ceph-mon[63804]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 162 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:26:57.480 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:26:57 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-17T00:26:57.480 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:26:57 smithi002 ceph-mon[63804]: from='mon.1 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-17T00:26:57.480 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:26:57 smithi002 ceph-mon[63804]: Upgrade: osd.3 is safe to restart 2024-09-17T00:26:57.480 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:26:57 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T00:26:57.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:26:57 smithi099 ceph-mon[56712]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 162 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:26:57.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:26:57 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-17T00:26:57.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:26:57 smithi099 ceph-mon[56712]: from='mon.1 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-17T00:26:57.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:26:57 smithi099 ceph-mon[56712]: Upgrade: osd.3 is safe to restart 2024-09-17T00:26:57.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:26:57 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T00:26:57.828 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-09-17T00:26:57.828 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": { 2024-09-17T00:26:57.828 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 19.1.1-330-gc2ddf972 (c2ddf9721ef254645201cdbabd9c8f49c30a586f) squid (stable)": 3 2024-09-17T00:26:57.828 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-09-17T00:26:57.829 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": { 2024-09-17T00:26:57.829 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 19.1.1-330-gc2ddf972 (c2ddf9721ef254645201cdbabd9c8f49c30a586f) squid (stable)": 2 2024-09-17T00:26:57.829 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-09-17T00:26:57.829 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": { 2024-09-17T00:26:57.829 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 5, 2024-09-17T00:26:57.829 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 19.1.1-330-gc2ddf972 (c2ddf9721ef254645201cdbabd9c8f49c30a586f) squid (stable)": 3 2024-09-17T00:26:57.829 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-09-17T00:26:57.829 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw": { 2024-09-17T00:26:57.829 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-09-17T00:26:57.829 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-09-17T00:26:57.829 INFO:teuthology.orchestra.run.smithi002.stdout: "overall": { 2024-09-17T00:26:57.829 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 9, 2024-09-17T00:26:57.829 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 19.1.1-330-gc2ddf972 (c2ddf9721ef254645201cdbabd9c8f49c30a586f) squid (stable)": 8 2024-09-17T00:26:57.830 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-09-17T00:26:57.830 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-09-17T00:26:58.184 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-09-17T00:26:58.184 INFO:teuthology.orchestra.run.smithi002.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f", 2024-09-17T00:26:58.185 INFO:teuthology.orchestra.run.smithi002.stdout: "in_progress": true, 2024-09-17T00:26:58.185 INFO:teuthology.orchestra.run.smithi002.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-17T00:26:58.185 INFO:teuthology.orchestra.run.smithi002.stdout: "services_complete": [ 2024-09-17T00:26:58.185 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr", 2024-09-17T00:26:58.185 INFO:teuthology.orchestra.run.smithi002.stdout: "mon" 2024-09-17T00:26:58.185 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-09-17T00:26:58.185 INFO:teuthology.orchestra.run.smithi002.stdout: "progress": "8/23 daemons upgraded", 2024-09-17T00:26:58.185 INFO:teuthology.orchestra.run.smithi002.stdout: "message": "Currently upgrading osd daemons", 2024-09-17T00:26:58.185 INFO:teuthology.orchestra.run.smithi002.stdout: "is_paused": false 2024-09-17T00:26:58.185 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-09-17T00:26:58.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:26:58 smithi002 ceph-mon[63804]: from='client.44476 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:26:58.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:26:58 smithi002 ceph-mon[63804]: from='client.34578 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:26:58.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:26:58 smithi002 ceph-mon[63804]: from='client.34584 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:26:58.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:26:58 smithi002 ceph-mon[63804]: from='client.? 172.21.15.2:0/4095793209' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:26:58.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:26:58 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:26:58.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:26:58 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-09-17T00:26:58.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:26:58 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:26:58.480 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:26:58 smithi002 ceph-mon[66732]: from='client.44476 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:26:58.480 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:26:58 smithi002 ceph-mon[66732]: from='client.34578 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:26:58.480 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:26:58 smithi002 ceph-mon[66732]: from='client.34584 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:26:58.480 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:26:58 smithi002 ceph-mon[66732]: from='client.? 172.21.15.2:0/4095793209' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:26:58.480 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:26:58 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:26:58.480 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:26:58 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-09-17T00:26:58.480 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:26:58 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:26:58.611 INFO:teuthology.orchestra.run.smithi002.stdout:HEALTH_OK 2024-09-17T00:26:58.656 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:26:58 smithi099 ceph-mon[56712]: from='client.44476 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:26:58.657 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:26:58 smithi099 ceph-mon[56712]: from='client.34578 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:26:58.657 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:26:58 smithi099 ceph-mon[56712]: from='client.34584 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:26:58.657 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:26:58 smithi099 ceph-mon[56712]: from='client.? 172.21.15.2:0/4095793209' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:26:58.657 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:26:58 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:26:58.657 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:26:58 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-09-17T00:26:58.657 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:26:58 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:26:58.967 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:26:58 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[52757]: ts=2024-09-17T00:26:58.659Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd name=CephOSDFlapping index=13 msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.1.1-330-gc2ddf972 (c2ddf9721ef254645201cdbabd9c8f49c30a586f) squid (stable)\", cluster=\"3aad27a2-7489-11ef-bceb-c7b262605968\", cluster_addr=\"172.21.15.2\", device_class=\"ssd\", hostname=\"smithi002\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.2\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.2\", device_class=\"ssd\", hostname=\"smithi002\", instance=\"172.21.15.99:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.2\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-17T00:26:59.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:26:59 smithi002 ceph-mon[66732]: Upgrade: Updating osd.3 2024-09-17T00:26:59.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:26:59 smithi002 ceph-mon[66732]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-09-17T00:26:59.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:26:59 smithi002 ceph-mon[66732]: Deploying daemon osd.3 on smithi002 2024-09-17T00:26:59.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:26:59 smithi002 ceph-mon[66732]: from='client.34596 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:26:59.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:26:59 smithi002 ceph-mon[66732]: from='client.? 172.21.15.2:0/184824374' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T00:26:59.480 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:26:59 smithi002 ceph-mon[63804]: Upgrade: Updating osd.3 2024-09-17T00:26:59.480 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:26:59 smithi002 ceph-mon[63804]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-09-17T00:26:59.480 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:26:59 smithi002 ceph-mon[63804]: Deploying daemon osd.3 on smithi002 2024-09-17T00:26:59.480 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:26:59 smithi002 ceph-mon[63804]: from='client.34596 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:26:59.480 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:26:59 smithi002 ceph-mon[63804]: from='client.? 172.21.15.2:0/184824374' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T00:26:59.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:26:59 smithi099 ceph-mon[56712]: Upgrade: Updating osd.3 2024-09-17T00:26:59.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:26:59 smithi099 ceph-mon[56712]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-09-17T00:26:59.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:26:59 smithi099 ceph-mon[56712]: Deploying daemon osd.3 on smithi002 2024-09-17T00:26:59.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:26:59 smithi099 ceph-mon[56712]: from='client.34596 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:26:59.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:26:59 smithi099 ceph-mon[56712]: from='client.? 172.21.15.2:0/184824374' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T00:27:01.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:27:01 smithi002 ceph-mon[66732]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-09-17T00:27:01.480 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:27:01 smithi002 ceph-mon[63804]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-09-17T00:27:01.480 INFO:journalctl@ceph.osd.3.smithi002.stdout:Sep 17 00:27:01 smithi002 systemd[1]: Stopping Ceph osd.3 for 3aad27a2-7489-11ef-bceb-c7b262605968... 2024-09-17T00:27:01.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:27:01 smithi099 ceph-mon[56712]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-09-17T00:27:01.979 INFO:journalctl@ceph.osd.3.smithi002.stdout:Sep 17 00:27:01 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-3[39814]: 2024-09-17T00:27:01.520+0000 7f68d0927700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-17T00:27:01.979 INFO:journalctl@ceph.osd.3.smithi002.stdout:Sep 17 00:27:01 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-3[39814]: 2024-09-17T00:27:01.520+0000 7f68d0927700 -1 osd.3 127 *** Got signal Terminated *** 2024-09-17T00:27:01.980 INFO:journalctl@ceph.osd.3.smithi002.stdout:Sep 17 00:27:01 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-3[39814]: 2024-09-17T00:27:01.520+0000 7f68d0927700 -1 osd.3 127 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-17T00:27:02.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:27:02 smithi002 ceph-mon[63804]: osd.3 marked itself down and dead 2024-09-17T00:27:02.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:27:02 smithi002 ceph-mon[66732]: osd.3 marked itself down and dead 2024-09-17T00:27:02.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:27:02 smithi099 ceph-mon[56712]: osd.3 marked itself down and dead 2024-09-17T00:27:02.979 INFO:journalctl@ceph.osd.3.smithi002.stdout:Sep 17 00:27:02 smithi002 podman[79647]: 2024-09-17 00:27:02.576146226 +0000 UTC m=+1.162567551 container died fd99d37644125bdbd78da56de0cffabb6503896eeba719d392cde1811fe0a822 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-3, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, RELEASE=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.19.8, vendor=Red Hat, Inc., architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.component=centos-stream-container, build-date=2022-05-03T08:36:31.336870, distribution-scope=public, vcs-type=git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, ceph=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, release=754, CEPH_POINT_RELEASE=-17.2.0, io.k8s.display-name=CentOS Stream 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.expose-services=, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, version=8, name=centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.tags=base centos centos-stream, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-09-17T00:27:03.352 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:27:03 smithi002 ceph-mon[66732]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-09-17T00:27:03.352 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:27:03 smithi002 ceph-mon[66732]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-17T00:27:03.352 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:27:03 smithi002 ceph-mon[66732]: osdmap e128: 8 total, 7 up, 8 in 2024-09-17T00:27:03.352 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:27:03 smithi002 ceph-mon[63804]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-09-17T00:27:03.353 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:27:03 smithi002 ceph-mon[63804]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-17T00:27:03.353 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:27:03 smithi002 ceph-mon[63804]: osdmap e128: 8 total, 7 up, 8 in 2024-09-17T00:27:03.353 INFO:journalctl@ceph.osd.3.smithi002.stdout:Sep 17 00:27:03 smithi002 podman[79647]: 2024-09-17 00:27:03.070317837 +0000 UTC m=+1.656739158 container cleanup fd99d37644125bdbd78da56de0cffabb6503896eeba719d392cde1811fe0a822 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-3, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, distribution-scope=public, io.buildah.version=1.19.8, GIT_CLEAN=True, com.redhat.component=centos-stream-container, maintainer=Guillaume Abrioux , summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, release=754, build-date=2022-05-03T08:36:31.336870, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-type=git, vendor=Red Hat, Inc., io.openshift.tags=base centos centos-stream, RELEASE=HEAD, ceph=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, name=centos-stream, GIT_BRANCH=HEAD, io.k8s.display-name=CentOS Stream 8, GIT_REPO=https://github.com/ceph/ceph-container.git, architecture=x86_64, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., CEPH_POINT_RELEASE=-17.2.0, version=8) 2024-09-17T00:27:03.353 INFO:journalctl@ceph.osd.3.smithi002.stdout:Sep 17 00:27:03 smithi002 bash[79647]: ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-3 2024-09-17T00:27:03.503 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:27:03 smithi099 ceph-mon[56712]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-09-17T00:27:03.503 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:27:03 smithi099 ceph-mon[56712]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-17T00:27:03.503 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:27:03 smithi099 ceph-mon[56712]: osdmap e128: 8 total, 7 up, 8 in 2024-09-17T00:27:03.678 INFO:journalctl@ceph.osd.3.smithi002.stdout:Sep 17 00:27:03 smithi002 podman[79663]: 2024-09-17 00:27:03.355050555 +0000 UTC m=+0.776351917 container remove fd99d37644125bdbd78da56de0cffabb6503896eeba719d392cde1811fe0a822 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-3, name=centos-stream, CEPH_POINT_RELEASE=-17.2.0, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base centos centos-stream, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, release=754, GIT_BRANCH=HEAD, io.openshift.expose-services=, io.buildah.version=1.19.8, RELEASE=HEAD, vendor=Red Hat, Inc., vcs-type=git, version=8, io.k8s.display-name=CentOS Stream 8, maintainer=Guillaume Abrioux , vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, distribution-scope=public, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.component=centos-stream-container, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, ceph=True, build-date=2022-05-03T08:36:31.336870) 2024-09-17T00:27:03.940 INFO:journalctl@ceph.osd.3.smithi002.stdout:Sep 17 00:27:03 smithi002 podman[79751]: 2024-09-17 00:27:03.584283932 +0000 UTC m=+0.016305554 image pull ebb0615149def4a3179c82c0cc47af75808300abb9f1142788e5e605a7ac0e5b quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f 2024-09-17T00:27:03.940 INFO:journalctl@ceph.osd.3.smithi002.stdout:Sep 17 00:27:03 smithi002 podman[79751]: 2024-09-17 00:27:03.706410865 +0000 UTC m=+0.138432519 container create 8a3368076194155c1baf37635c5d57ce29b5e34575dcbef82c8c7f803965d9f4 (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-3-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, RELEASE=squid-c2ddf97, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-09-17T00:27:03.940 INFO:journalctl@ceph.osd.3.smithi002.stdout:Sep 17 00:27:03 smithi002 podman[79751]: 2024-09-17 00:27:03.942426057 +0000 UTC m=+0.374447683 container init 8a3368076194155c1baf37635c5d57ce29b5e34575dcbef82c8c7f803965d9f4 (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-3-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-c2ddf97, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-09-17T00:27:03.967 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:27:03 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[52757]: ts=2024-09-17T00:27:03.506Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes name=CephNodeDiskspaceWarning index=4 msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi002\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi002\", job=\"node\", machine=\"x86_64\", nodename=\"smithi002\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"3aad27a2-7489-11ef-bceb-c7b262605968\", domainname=\"(none)\", instance=\"smithi002\", job=\"node\", machine=\"x86_64\", nodename=\"smithi002\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-17T00:27:04.229 INFO:journalctl@ceph.osd.3.smithi002.stdout:Sep 17 00:27:03 smithi002 podman[79751]: 2024-09-17 00:27:03.946486964 +0000 UTC m=+0.378508588 container start 8a3368076194155c1baf37635c5d57ce29b5e34575dcbef82c8c7f803965d9f4 (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-3-deactivate, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, RELEASE=squid-c2ddf97, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-09-17T00:27:04.229 INFO:journalctl@ceph.osd.3.smithi002.stdout:Sep 17 00:27:04 smithi002 podman[79751]: 2024-09-17 00:27:04.033757819 +0000 UTC m=+0.465779430 container attach 8a3368076194155c1baf37635c5d57ce29b5e34575dcbef82c8c7f803965d9f4 (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-3-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , RELEASE=squid-c2ddf97, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-09-17T00:27:04.229 INFO:journalctl@ceph.osd.3.smithi002.stdout:Sep 17 00:27:04 smithi002 podman[79751]: 2024-09-17 00:27:04.147299495 +0000 UTC m=+0.579321119 container died 8a3368076194155c1baf37635c5d57ce29b5e34575dcbef82c8c7f803965d9f4 (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-3-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=squid-c2ddf97, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD) 2024-09-17T00:27:04.501 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:27:04 smithi002 ceph-mon[63804]: from='client.34557 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:27:04.501 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:27:04 smithi002 ceph-mon[63804]: osdmap e129: 8 total, 7 up, 8 in 2024-09-17T00:27:04.501 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:27:04 smithi002 ceph-mon[66732]: from='client.34557 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:27:04.501 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:27:04 smithi002 ceph-mon[66732]: osdmap e129: 8 total, 7 up, 8 in 2024-09-17T00:27:04.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:27:04 smithi099 ceph-mon[56712]: from='client.34557 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:27:04.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:27:04 smithi099 ceph-mon[56712]: osdmap e129: 8 total, 7 up, 8 in 2024-09-17T00:27:05.035 INFO:journalctl@ceph.osd.3.smithi002.stdout:Sep 17 00:27:04 smithi002 podman[79774]: 2024-09-17 00:27:04.787253477 +0000 UTC m=+0.633442396 container remove 8a3368076194155c1baf37635c5d57ce29b5e34575dcbef82c8c7f803965d9f4 (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-3-deactivate, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-c2ddf97, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-09-17T00:27:05.035 INFO:journalctl@ceph.osd.3.smithi002.stdout:Sep 17 00:27:04 smithi002 systemd[1]: ceph-3aad27a2-7489-11ef-bceb-c7b262605968@osd.3.service: Deactivated successfully. 2024-09-17T00:27:05.035 INFO:journalctl@ceph.osd.3.smithi002.stdout:Sep 17 00:27:04 smithi002 systemd[1]: Stopped Ceph osd.3 for 3aad27a2-7489-11ef-bceb-c7b262605968. 2024-09-17T00:27:05.035 INFO:journalctl@ceph.osd.3.smithi002.stdout:Sep 17 00:27:04 smithi002 systemd[1]: ceph-3aad27a2-7489-11ef-bceb-c7b262605968@osd.3.service: Consumed 9.506s CPU time. 2024-09-17T00:27:05.035 INFO:journalctl@ceph.osd.3.smithi002.stdout:Sep 17 00:27:05 smithi002 systemd[1]: Starting Ceph osd.3 for 3aad27a2-7489-11ef-bceb-c7b262605968... 2024-09-17T00:27:05.530 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:27:05 smithi002 ceph-mon[63804]: pgmap v130: 161 pgs: 33 peering, 13 stale+active+clean, 115 active+clean; 585 KiB data, 162 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:27:05.530 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:27:05 smithi002 ceph-mon[63804]: Health check failed: Reduced data availability: 7 pgs inactive, 13 pgs peering (PG_AVAILABILITY) 2024-09-17T00:27:05.530 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:27:05 smithi002 ceph-mon[66732]: pgmap v130: 161 pgs: 33 peering, 13 stale+active+clean, 115 active+clean; 585 KiB data, 162 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:27:05.530 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:27:05 smithi002 ceph-mon[66732]: Health check failed: Reduced data availability: 7 pgs inactive, 13 pgs peering (PG_AVAILABILITY) 2024-09-17T00:27:05.530 INFO:journalctl@ceph.osd.3.smithi002.stdout:Sep 17 00:27:05 smithi002 podman[79875]: 2024-09-17 00:27:05.151869152 +0000 UTC m=+0.016089260 image pull ebb0615149def4a3179c82c0cc47af75808300abb9f1142788e5e605a7ac0e5b quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f 2024-09-17T00:27:05.530 INFO:journalctl@ceph.osd.3.smithi002.stdout:Sep 17 00:27:05 smithi002 podman[79875]: 2024-09-17 00:27:05.263715402 +0000 UTC m=+0.127935536 container create 35b67487bdec55d4b96912415d3b1d393ed1ad8e4db1873ec744a70d457900bb (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-3-activate, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240731, RELEASE=squid-c2ddf97, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-17T00:27:05.530 INFO:journalctl@ceph.osd.3.smithi002.stdout:Sep 17 00:27:05 smithi002 podman[79875]: 2024-09-17 00:27:05.533058822 +0000 UTC m=+0.397278925 container init 35b67487bdec55d4b96912415d3b1d393ed1ad8e4db1873ec744a70d457900bb (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-3-activate, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=squid-c2ddf97, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True) 2024-09-17T00:27:05.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:27:05 smithi099 ceph-mon[56712]: pgmap v130: 161 pgs: 33 peering, 13 stale+active+clean, 115 active+clean; 585 KiB data, 162 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:27:05.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:27:05 smithi099 ceph-mon[56712]: Health check failed: Reduced data availability: 7 pgs inactive, 13 pgs peering (PG_AVAILABILITY) 2024-09-17T00:27:05.979 INFO:journalctl@ceph.osd.3.smithi002.stdout:Sep 17 00:27:05 smithi002 podman[79875]: 2024-09-17 00:27:05.536917375 +0000 UTC m=+0.401137464 container start 35b67487bdec55d4b96912415d3b1d393ed1ad8e4db1873ec744a70d457900bb (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-3-activate, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , RELEASE=squid-c2ddf97, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-17T00:27:05.979 INFO:journalctl@ceph.osd.3.smithi002.stdout:Sep 17 00:27:05 smithi002 podman[79875]: 2024-09-17 00:27:05.616267232 +0000 UTC m=+0.480487338 container attach 35b67487bdec55d4b96912415d3b1d393ed1ad8e4db1873ec744a70d457900bb (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-3-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, RELEASE=squid-c2ddf97, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.build-date=20240731) 2024-09-17T00:27:05.980 INFO:journalctl@ceph.osd.3.smithi002.stdout:Sep 17 00:27:05 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-3-activate[79890]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-17T00:27:05.980 INFO:journalctl@ceph.osd.3.smithi002.stdout:Sep 17 00:27:05 smithi002 bash[79875]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-17T00:27:05.980 INFO:journalctl@ceph.osd.3.smithi002.stdout:Sep 17 00:27:05 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-3-activate[79890]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-17T00:27:05.980 INFO:journalctl@ceph.osd.3.smithi002.stdout:Sep 17 00:27:05 smithi002 bash[79875]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-17T00:27:06.431 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:27:06 smithi002 ceph-mon[63804]: Health check failed: Degraded data redundancy: 4/726 objects degraded (0.551%), 3 pgs degraded (PG_DEGRADED) 2024-09-17T00:27:06.432 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:27:06 smithi002 ceph-mon[66732]: Health check failed: Degraded data redundancy: 4/726 objects degraded (0.551%), 3 pgs degraded (PG_DEGRADED) 2024-09-17T00:27:06.432 INFO:journalctl@ceph.osd.3.smithi002.stdout:Sep 17 00:27:06 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-3-activate[79890]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-17T00:27:06.432 INFO:journalctl@ceph.osd.3.smithi002.stdout:Sep 17 00:27:06 smithi002 bash[79875]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-17T00:27:06.432 INFO:journalctl@ceph.osd.3.smithi002.stdout:Sep 17 00:27:06 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-3-activate[79890]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-09-17T00:27:06.432 INFO:journalctl@ceph.osd.3.smithi002.stdout:Sep 17 00:27:06 smithi002 bash[79875]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-09-17T00:27:06.432 INFO:journalctl@ceph.osd.3.smithi002.stdout:Sep 17 00:27:06 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-3-activate[79890]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-09-17T00:27:06.433 INFO:journalctl@ceph.osd.3.smithi002.stdout:Sep 17 00:27:06 smithi002 bash[79875]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-09-17T00:27:06.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:27:06 smithi099 ceph-mon[56712]: Health check failed: Degraded data redundancy: 4/726 objects degraded (0.551%), 3 pgs degraded (PG_DEGRADED) 2024-09-17T00:27:06.729 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:27:06 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: ::ffff:172.21.15.99 - - [17/Sep/2024:00:27:06] "GET /metrics HTTP/1.1" 200 38436 "" "Prometheus/2.43.0" 2024-09-17T00:27:06.729 INFO:journalctl@ceph.osd.3.smithi002.stdout:Sep 17 00:27:06 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-3-activate[79890]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-09-17T00:27:06.729 INFO:journalctl@ceph.osd.3.smithi002.stdout:Sep 17 00:27:06 smithi002 bash[79875]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-09-17T00:27:06.729 INFO:journalctl@ceph.osd.3.smithi002.stdout:Sep 17 00:27:06 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-3-activate[79890]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-3/block 2024-09-17T00:27:06.730 INFO:journalctl@ceph.osd.3.smithi002.stdout:Sep 17 00:27:06 smithi002 bash[79875]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-3/block 2024-09-17T00:27:06.730 INFO:journalctl@ceph.osd.3.smithi002.stdout:Sep 17 00:27:06 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-3-activate[79890]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-17T00:27:06.730 INFO:journalctl@ceph.osd.3.smithi002.stdout:Sep 17 00:27:06 smithi002 bash[79875]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-17T00:27:06.730 INFO:journalctl@ceph.osd.3.smithi002.stdout:Sep 17 00:27:06 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-3-activate[79890]: --> ceph-volume raw activate successful for osd ID: 3 2024-09-17T00:27:06.730 INFO:journalctl@ceph.osd.3.smithi002.stdout:Sep 17 00:27:06 smithi002 bash[79875]: --> ceph-volume raw activate successful for osd ID: 3 2024-09-17T00:27:06.730 INFO:journalctl@ceph.osd.3.smithi002.stdout:Sep 17 00:27:06 smithi002 podman[79875]: 2024-09-17 00:27:06.464311818 +0000 UTC m=+1.328531914 container died 35b67487bdec55d4b96912415d3b1d393ed1ad8e4db1873ec744a70d457900bb (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-3-activate, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=squid-c2ddf97, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-09-17T00:27:07.446 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:27:07 smithi002 ceph-mon[66732]: pgmap v131: 161 pgs: 4 active+undersized, 33 peering, 11 stale+active+clean, 3 active+undersized+degraded, 110 active+clean; 585 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 4/726 objects degraded (0.551%) 2024-09-17T00:27:07.447 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:27:07 smithi002 ceph-mon[63804]: pgmap v131: 161 pgs: 4 active+undersized, 33 peering, 11 stale+active+clean, 3 active+undersized+degraded, 110 active+clean; 585 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 4/726 objects degraded (0.551%) 2024-09-17T00:27:07.447 INFO:journalctl@ceph.osd.3.smithi002.stdout:Sep 17 00:27:07 smithi002 podman[80035]: 2024-09-17 00:27:07.111837184 +0000 UTC m=+0.641057303 container remove 35b67487bdec55d4b96912415d3b1d393ed1ad8e4db1873ec744a70d457900bb (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-3-activate, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=squid-c2ddf97, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-17T00:27:07.447 INFO:journalctl@ceph.osd.3.smithi002.stdout:Sep 17 00:27:07 smithi002 podman[80094]: 2024-09-17 00:27:07.231903312 +0000 UTC m=+0.017040381 image pull ebb0615149def4a3179c82c0cc47af75808300abb9f1142788e5e605a7ac0e5b quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f 2024-09-17T00:27:07.447 INFO:journalctl@ceph.osd.3.smithi002.stdout:Sep 17 00:27:07 smithi002 podman[80094]: 2024-09-17 00:27:07.32933841 +0000 UTC m=+0.114475475 container create 980f2a25d096db44b54da22ac4e7c1587a0533a4815afd6f1769a6f3ea3f1a22 (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-3, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, RELEASE=squid-c2ddf97, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-09-17T00:27:07.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:27:07 smithi099 ceph-mon[56712]: pgmap v131: 161 pgs: 4 active+undersized, 33 peering, 11 stale+active+clean, 3 active+undersized+degraded, 110 active+clean; 585 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 4/726 objects degraded (0.551%) 2024-09-17T00:27:07.729 INFO:journalctl@ceph.osd.3.smithi002.stdout:Sep 17 00:27:07 smithi002 podman[80094]: 2024-09-17 00:27:07.5737574 +0000 UTC m=+0.358894466 container init 980f2a25d096db44b54da22ac4e7c1587a0533a4815afd6f1769a6f3ea3f1a22 (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-3, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=squid-c2ddf97, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-09-17T00:27:07.729 INFO:journalctl@ceph.osd.3.smithi002.stdout:Sep 17 00:27:07 smithi002 podman[80094]: 2024-09-17 00:27:07.577705086 +0000 UTC m=+0.362842150 container start 980f2a25d096db44b54da22ac4e7c1587a0533a4815afd6f1769a6f3ea3f1a22 (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-3, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=squid-c2ddf97, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2) 2024-09-17T00:27:07.729 INFO:journalctl@ceph.osd.3.smithi002.stdout:Sep 17 00:27:07 smithi002 ceph-osd[80114]: -- 172.21.15.2:0/3218680881 <== mon.1 v2:172.21.15.2:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55ed7e1b1a40 con 0x55ed7ef78400 2024-09-17T00:27:07.729 INFO:journalctl@ceph.osd.3.smithi002.stdout:Sep 17 00:27:07 smithi002 bash[80094]: 980f2a25d096db44b54da22ac4e7c1587a0533a4815afd6f1769a6f3ea3f1a22 2024-09-17T00:27:07.729 INFO:journalctl@ceph.osd.3.smithi002.stdout:Sep 17 00:27:07 smithi002 systemd[1]: Started Ceph osd.3 for 3aad27a2-7489-11ef-bceb-c7b262605968. 2024-09-17T00:27:08.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:27:08 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:27:08.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:27:08 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:27:08.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:27:08 smithi099 ceph-mon[56712]: pgmap v132: 161 pgs: 40 active+undersized, 26 active+undersized+degraded, 95 active+clean; 585 KiB data, 163 MiB used, 715 GiB / 715 GiB avail; 97/726 objects degraded (13.361%) 2024-09-17T00:27:08.967 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:27:08 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[52757]: ts=2024-09-17T00:27:08.659Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd name=CephOSDFlapping index=13 msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.1.1-330-gc2ddf972 (c2ddf9721ef254645201cdbabd9c8f49c30a586f) squid (stable)\", cluster=\"3aad27a2-7489-11ef-bceb-c7b262605968\", cluster_addr=\"172.21.15.2\", device_class=\"ssd\", hostname=\"smithi002\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.2\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.2\", device_class=\"ssd\", hostname=\"smithi002\", instance=\"172.21.15.99:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.2\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-17T00:27:08.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:27:08 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:27:08.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:27:08 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:27:08.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:27:08 smithi002 ceph-mon[66732]: pgmap v132: 161 pgs: 40 active+undersized, 26 active+undersized+degraded, 95 active+clean; 585 KiB data, 163 MiB used, 715 GiB / 715 GiB avail; 97/726 objects degraded (13.361%) 2024-09-17T00:27:08.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:27:08 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:27:08.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:27:08 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:27:08.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:27:08 smithi002 ceph-mon[63804]: pgmap v132: 161 pgs: 40 active+undersized, 26 active+undersized+degraded, 95 active+clean; 585 KiB data, 163 MiB used, 715 GiB / 715 GiB avail; 97/726 objects degraded (13.361%) 2024-09-17T00:27:09.552 INFO:journalctl@ceph.osd.3.smithi002.stdout:Sep 17 00:27:09 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-3[80109]: 2024-09-17T00:27:09.477+0000 7f34b2766640 -1 Falling back to public interface 2024-09-17T00:27:09.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:27:09 smithi002 ceph-mon[63804]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 7 pgs inactive, 13 pgs peering) 2024-09-17T00:27:09.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:27:09 smithi002 ceph-mon[66732]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 7 pgs inactive, 13 pgs peering) 2024-09-17T00:27:10.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:27:09 smithi099 ceph-mon[56712]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 7 pgs inactive, 13 pgs peering) 2024-09-17T00:27:11.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:27:10 smithi099 ceph-mon[56712]: pgmap v133: 161 pgs: 40 active+undersized, 26 active+undersized+degraded, 95 active+clean; 585 KiB data, 163 MiB used, 715 GiB / 715 GiB avail; 97/726 objects degraded (13.361%) 2024-09-17T00:27:11.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:27:10 smithi002 ceph-mon[63804]: pgmap v133: 161 pgs: 40 active+undersized, 26 active+undersized+degraded, 95 active+clean; 585 KiB data, 163 MiB used, 715 GiB / 715 GiB avail; 97/726 objects degraded (13.361%) 2024-09-17T00:27:11.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:27:10 smithi002 ceph-mon[66732]: pgmap v133: 161 pgs: 40 active+undersized, 26 active+undersized+degraded, 95 active+clean; 585 KiB data, 163 MiB used, 715 GiB / 715 GiB avail; 97/726 objects degraded (13.361%) 2024-09-17T00:27:13.021 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:27:12 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:27:13.021 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:27:12 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T00:27:13.021 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:27:12 smithi002 ceph-mon[66732]: pgmap v134: 161 pgs: 40 active+undersized, 26 active+undersized+degraded, 95 active+clean; 585 KiB data, 163 MiB used, 715 GiB / 715 GiB avail; 97/726 objects degraded (13.361%) 2024-09-17T00:27:13.022 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:27:12 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:27:13.022 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:27:12 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:27:13.022 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:27:12 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:27:13.022 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:27:12 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T00:27:13.023 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:27:12 smithi002 ceph-mon[63804]: pgmap v134: 161 pgs: 40 active+undersized, 26 active+undersized+degraded, 95 active+clean; 585 KiB data, 163 MiB used, 715 GiB / 715 GiB avail; 97/726 objects degraded (13.361%) 2024-09-17T00:27:13.023 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:27:12 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:27:13.023 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:27:12 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:27:13.023 INFO:journalctl@ceph.osd.3.smithi002.stdout:Sep 17 00:27:12 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-3[80109]: 2024-09-17T00:27:12.753+0000 7f34b2766640 -1 osd.3 0 read_superblock omap replica is missing. 2024-09-17T00:27:13.023 INFO:journalctl@ceph.osd.3.smithi002.stdout:Sep 17 00:27:12 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-3[80109]: 2024-09-17T00:27:12.783+0000 7f34b2766640 -1 osd.3 127 log_to_monitors true 2024-09-17T00:27:13.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:27:12 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:27:13.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:27:12 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T00:27:13.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:27:12 smithi099 ceph-mon[56712]: pgmap v134: 161 pgs: 40 active+undersized, 26 active+undersized+degraded, 95 active+clean; 585 KiB data, 163 MiB used, 715 GiB / 715 GiB avail; 97/726 objects degraded (13.361%) 2024-09-17T00:27:13.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:27:12 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:27:13.218 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:27:12 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:27:13.764 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:27:13 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[52757]: ts=2024-09-17T00:27:13.505Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=nodes name=CephNodeDiskspaceWarning index=4 msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on (instance) group_left (nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi002\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi002\", job=\"node\", machine=\"x86_64\", nodename=\"smithi002\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"3aad27a2-7489-11ef-bceb-c7b262605968\", domainname=\"(none)\", instance=\"smithi002\", job=\"node\", machine=\"x86_64\", nodename=\"smithi002\", release=\"5.14.0-508.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-17T00:27:13.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:27:13 smithi002 ceph-mon[66732]: from='osd.3 [v2:172.21.15.2:6826/1186536034,v1:172.21.15.2:6827/1186536034]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-17T00:27:13.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:27:13 smithi002 ceph-mon[66732]: from='client.34557 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:27:13.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:27:13 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:27:13.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:27:13 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:27:13.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:27:13 smithi002 ceph-mon[66732]: Health check update: Degraded data redundancy: 97/726 objects degraded (13.361%), 26 pgs degraded (PG_DEGRADED) 2024-09-17T00:27:13.980 INFO:journalctl@ceph.osd.3.smithi002.stdout:Sep 17 00:27:13 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-3[80109]: 2024-09-17T00:27:13.791+0000 7f34aa511640 -1 osd.3 127 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T00:27:13.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:27:13 smithi002 ceph-mon[63804]: from='osd.3 [v2:172.21.15.2:6826/1186536034,v1:172.21.15.2:6827/1186536034]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-17T00:27:13.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:27:13 smithi002 ceph-mon[63804]: from='client.34557 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:27:13.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:27:13 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:27:13.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:27:13 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:27:13.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:27:13 smithi002 ceph-mon[63804]: Health check update: Degraded data redundancy: 97/726 objects degraded (13.361%), 26 pgs degraded (PG_DEGRADED) 2024-09-17T00:27:14.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:27:13 smithi099 ceph-mon[56712]: from='osd.3 [v2:172.21.15.2:6826/1186536034,v1:172.21.15.2:6827/1186536034]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-17T00:27:14.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:27:13 smithi099 ceph-mon[56712]: from='client.34557 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:27:14.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:27:13 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:27:14.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:27:13 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:27:14.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:27:13 smithi099 ceph-mon[56712]: Health check update: Degraded data redundancy: 97/726 objects degraded (13.361%), 26 pgs degraded (PG_DEGRADED) 2024-09-17T00:27:15.025 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:27:14 smithi002 ceph-mon[66732]: from='osd.3 [v2:172.21.15.2:6826/1186536034,v1:172.21.15.2:6827/1186536034]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-17T00:27:15.025 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:27:14 smithi002 ceph-mon[66732]: osdmap e130: 8 total, 7 up, 8 in 2024-09-17T00:27:15.025 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:27:14 smithi002 ceph-mon[66732]: from='osd.3 [v2:172.21.15.2:6826/1186536034,v1:172.21.15.2:6827/1186536034]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]: dispatch 2024-09-17T00:27:15.025 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:27:14 smithi002 ceph-mon[66732]: pgmap v136: 161 pgs: 40 active+undersized, 26 active+undersized+degraded, 95 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 409 B/s rd, 0 op/s; 97/726 objects degraded (13.361%) 2024-09-17T00:27:15.026 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:27:14 smithi002 ceph-mon[63804]: from='osd.3 [v2:172.21.15.2:6826/1186536034,v1:172.21.15.2:6827/1186536034]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-17T00:27:15.026 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:27:14 smithi002 ceph-mon[63804]: osdmap e130: 8 total, 7 up, 8 in 2024-09-17T00:27:15.026 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:27:14 smithi002 ceph-mon[63804]: from='osd.3 [v2:172.21.15.2:6826/1186536034,v1:172.21.15.2:6827/1186536034]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]: dispatch 2024-09-17T00:27:15.026 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:27:14 smithi002 ceph-mon[63804]: pgmap v136: 161 pgs: 40 active+undersized, 26 active+undersized+degraded, 95 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 409 B/s rd, 0 op/s; 97/726 objects degraded (13.361%) 2024-09-17T00:27:15.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:27:14 smithi099 ceph-mon[56712]: from='osd.3 [v2:172.21.15.2:6826/1186536034,v1:172.21.15.2:6827/1186536034]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-17T00:27:15.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:27:14 smithi099 ceph-mon[56712]: osdmap e130: 8 total, 7 up, 8 in 2024-09-17T00:27:15.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:27:14 smithi099 ceph-mon[56712]: from='osd.3 [v2:172.21.15.2:6826/1186536034,v1:172.21.15.2:6827/1186536034]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi002", "root=default"]}]: dispatch 2024-09-17T00:27:15.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:27:14 smithi099 ceph-mon[56712]: pgmap v136: 161 pgs: 40 active+undersized, 26 active+undersized+degraded, 95 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 409 B/s rd, 0 op/s; 97/726 objects degraded (13.361%) 2024-09-17T00:27:16.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:27:15 smithi099 ceph-mon[56712]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-17T00:27:16.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:27:15 smithi099 ceph-mon[56712]: osd.3 [v2:172.21.15.2:6826/1186536034,v1:172.21.15.2:6827/1186536034] boot 2024-09-17T00:27:16.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:27:15 smithi099 ceph-mon[56712]: osdmap e131: 8 total, 8 up, 8 in 2024-09-17T00:27:16.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:27:15 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T00:27:16.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:27:15 smithi099 ceph-mon[56712]: osdmap e132: 8 total, 8 up, 8 in 2024-09-17T00:27:16.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:27:15 smithi002 ceph-mon[66732]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-17T00:27:16.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:27:15 smithi002 ceph-mon[66732]: osd.3 [v2:172.21.15.2:6826/1186536034,v1:172.21.15.2:6827/1186536034] boot 2024-09-17T00:27:16.230 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:27:15 smithi002 ceph-mon[66732]: osdmap e131: 8 total, 8 up, 8 in 2024-09-17T00:27:16.230 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:27:15 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T00:27:16.230 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:27:15 smithi002 ceph-mon[66732]: osdmap e132: 8 total, 8 up, 8 in 2024-09-17T00:27:16.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:27:15 smithi002 ceph-mon[63804]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-17T00:27:16.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:27:15 smithi002 ceph-mon[63804]: osd.3 [v2:172.21.15.2:6826/1186536034,v1:172.21.15.2:6827/1186536034] boot 2024-09-17T00:27:16.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:27:15 smithi002 ceph-mon[63804]: osdmap e131: 8 total, 8 up, 8 in 2024-09-17T00:27:16.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:27:15 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-17T00:27:16.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:27:15 smithi002 ceph-mon[63804]: osdmap e132: 8 total, 8 up, 8 in 2024-09-17T00:27:16.979 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:27:16 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: ::ffff:172.21.15.99 - - [17/Sep/2024:00:27:16] "GET /metrics HTTP/1.1" 200 38438 "" "Prometheus/2.43.0" 2024-09-17T00:27:17.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:27:17 smithi002 ceph-mon[63804]: pgmap v139: 161 pgs: 40 active+undersized, 26 active+undersized+degraded, 95 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 97/726 objects degraded (13.361%) 2024-09-17T00:27:17.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:27:17 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:27:17.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:27:17 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:27:17.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:27:17 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:27:17.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:27:17 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:27:17.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:27:17 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:27:17.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:27:17 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi002.vmouem", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-17T00:27:17.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:27:17 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi002.vmouem", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-17T00:27:17.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:27:17 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:27:17.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:27:17 smithi002 ceph-mon[66732]: pgmap v139: 161 pgs: 40 active+undersized, 26 active+undersized+degraded, 95 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 97/726 objects degraded (13.361%) 2024-09-17T00:27:17.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:27:17 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:27:17.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:27:17 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:27:17.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:27:17 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:27:17.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:27:17 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:27:17.731 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:27:17 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:27:17.731 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:27:17 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi002.vmouem", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-17T00:27:17.731 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:27:17 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi002.vmouem", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-17T00:27:17.731 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:27:17 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:27:17.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:27:17 smithi099 ceph-mon[56712]: pgmap v139: 161 pgs: 40 active+undersized, 26 active+undersized+degraded, 95 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 97/726 objects degraded (13.361%) 2024-09-17T00:27:17.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:27:17 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:27:17.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:27:17 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:27:17.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:27:17 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:27:17.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:27:17 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:27:17.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:27:17 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:27:17.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:27:17 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi002.vmouem", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-17T00:27:17.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:27:17 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi002.vmouem", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-17T00:27:17.968 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:27:17 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:27:18.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:27:18 smithi099 ceph-mon[56712]: Reconfiguring iscsi.foo.smithi002.vmouem (dependencies changed)... 2024-09-17T00:27:18.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:27:18 smithi099 ceph-mon[56712]: Reconfiguring daemon iscsi.foo.smithi002.vmouem on smithi002 2024-09-17T00:27:18.967 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:27:18 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[52757]: ts=2024-09-17T00:27:18.659Z caller=alerting.go:366 level=warn component="rule manager" alert="unsupported value type" msg="Expanding alert template failed" err="error executing template __alert_CephOSDDown: template: __alert_CephOSDDown:1:358: executing \"__alert_CephOSDDown\" at : error calling query: found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.1.1-330-gc2ddf972 (c2ddf9721ef254645201cdbabd9c8f49c30a586f) squid (stable)\", cluster=\"3aad27a2-7489-11ef-bceb-c7b262605968\", cluster_addr=\"172.21.15.2\", device_class=\"ssd\", hostname=\"smithi002\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.2\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.2\", device_class=\"ssd\", hostname=\"smithi002\", instance=\"172.21.15.99:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.2\"}];many-to-many matching not allowed: matching labels must be unique on one side" data="unsupported value type" 2024-09-17T00:27:18.967 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:27:18 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[52757]: ts=2024-09-17T00:27:18.660Z caller=manager.go:640 level=warn component="rule manager" file=/etc/prometheus/alerting/ceph_alerts.yml group=osd name=CephOSDFlapping index=13 msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on (ceph_daemon) group_left (hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.1.1-330-gc2ddf972 (c2ddf9721ef254645201cdbabd9c8f49c30a586f) squid (stable)\", cluster=\"3aad27a2-7489-11ef-bceb-c7b262605968\", cluster_addr=\"172.21.15.2\", device_class=\"ssd\", hostname=\"smithi002\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.2\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.2\", device_class=\"ssd\", hostname=\"smithi002\", instance=\"172.21.15.99:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.2\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-09-17T00:27:18.968 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:27:18 smithi002 ceph-mon[66732]: Reconfiguring iscsi.foo.smithi002.vmouem (dependencies changed)... 2024-09-17T00:27:18.969 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:27:18 smithi002 ceph-mon[66732]: Reconfiguring daemon iscsi.foo.smithi002.vmouem on smithi002 2024-09-17T00:27:18.969 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:27:18 smithi002 ceph-mon[63804]: Reconfiguring iscsi.foo.smithi002.vmouem (dependencies changed)... 2024-09-17T00:27:18.969 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:27:18 smithi002 ceph-mon[63804]: Reconfiguring daemon iscsi.foo.smithi002.vmouem on smithi002 2024-09-17T00:27:19.831 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:27:19 smithi002 ceph-mon[66732]: pgmap v140: 161 pgs: 19 active+undersized, 13 active+undersized+degraded, 129 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 45/726 objects degraded (6.198%) 2024-09-17T00:27:19.831 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:27:19 smithi002 ceph-mon[66732]: Health check update: Degraded data redundancy: 45/726 objects degraded (6.198%), 13 pgs degraded (PG_DEGRADED) 2024-09-17T00:27:19.832 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:27:19 smithi002 ceph-mon[63804]: pgmap v140: 161 pgs: 19 active+undersized, 13 active+undersized+degraded, 129 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 45/726 objects degraded (6.198%) 2024-09-17T00:27:19.832 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:27:19 smithi002 ceph-mon[63804]: Health check update: Degraded data redundancy: 45/726 objects degraded (6.198%), 13 pgs degraded (PG_DEGRADED) 2024-09-17T00:27:19.832 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:27:19 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: 2024-09-17T00:27:19.684+0000 7f17e1e2b640 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi002.vmouem ... 2024-09-17T00:27:19.833 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:27:19 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: Creating ceph-iscsi config... 2024-09-17T00:27:19.833 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:27:19 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/iscsi-gateway.cfg 2024-09-17T00:27:19.833 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:27:19 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/tcmu-runner-entrypoint.sh 2024-09-17T00:27:19.833 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:27:19 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: Non-zero exit code 1 from systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem 2024-09-17T00:27:19.833 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:27:19 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: systemctl: stderr Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:27:19.833 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:27:19 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: systemctl: stderr See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:27:19.833 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:27:19 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: Traceback (most recent call last): 2024-09-17T00:27:19.833 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:27:19 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-17T00:27:19.833 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:27:19 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: return _run_code(code, main_globals, None, 2024-09-17T00:27:19.834 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:27:19 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-17T00:27:19.834 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:27:19 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: exec(code, run_globals) 2024-09-17T00:27:19.834 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:27:19 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5579, in 2024-09-17T00:27:19.834 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:27:19 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5567, in main 2024-09-17T00:27:19.834 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:27:19 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3049, in command_deploy_from 2024-09-17T00:27:19.834 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:27:19 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3084, in _common_deploy 2024-09-17T00:27:19.834 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:27:19 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3104, in _deploy_daemon_container 2024-09-17T00:27:19.834 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:27:19 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 1124, in deploy_daemon 2024-09-17T00:27:19.834 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:27:19 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/tmp/tmpnroy8d0w.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-17T00:27:19.835 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:27:19 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: RuntimeError: Failed command: systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem: Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:27:19.835 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:27:19 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:27:19.835 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:27:19 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: Traceback (most recent call last): 2024-09-17T00:27:19.835 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:27:19 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1134, in _check_daemons 2024-09-17T00:27:19.835 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:27:19 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: self.mgr._daemon_action(daemon_spec, action=action) 2024-09-17T00:27:19.835 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:27:19 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2473, in _daemon_action 2024-09-17T00:27:19.835 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:27:19 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: return self.wait_async( 2024-09-17T00:27:19.835 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:27:19 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/usr/share/ceph/mgr/cephadm/module.py", line 796, in wait_async 2024-09-17T00:27:19.835 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:27:19 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: return self.event_loop.get_result(coro, timeout) 2024-09-17T00:27:19.835 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:27:19 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-09-17T00:27:19.835 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:27:19 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: return future.result(timeout) 2024-09-17T00:27:19.835 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:27:19 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/lib64/python3.9/concurrent/futures/_base.py", line 446, in result 2024-09-17T00:27:19.835 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:27:19 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: return self.__get_result() 2024-09-17T00:27:19.836 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:27:19 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/lib64/python3.9/concurrent/futures/_base.py", line 391, in __get_result 2024-09-17T00:27:19.836 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:27:19 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: raise self._exception 2024-09-17T00:27:19.836 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:27:19 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1370, in _create_daemon 2024-09-17T00:27:19.836 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:27:19 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: out, err, code = await self._run_cephadm( 2024-09-17T00:27:19.836 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:27:19 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1713, in _run_cephadm 2024-09-17T00:27:19.836 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:27:19 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: raise OrchestratorError( 2024-09-17T00:27:19.836 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:27:19 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi002.vmouem ... 2024-09-17T00:27:19.836 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:27:19 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: Creating ceph-iscsi config... 2024-09-17T00:27:19.836 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:27:19 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/iscsi-gateway.cfg 2024-09-17T00:27:19.836 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:27:19 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/tcmu-runner-entrypoint.sh 2024-09-17T00:27:19.836 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:27:19 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: Non-zero exit code 1 from systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem 2024-09-17T00:27:19.836 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:27:19 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: systemctl: stderr Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:27:19.836 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:27:19 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: systemctl: stderr See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:27:19.837 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:27:19 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: Traceback (most recent call last): 2024-09-17T00:27:19.837 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:27:19 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-17T00:27:19.837 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:27:19 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: return _run_code(code, main_globals, None, 2024-09-17T00:27:19.837 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:27:19 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-17T00:27:19.837 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:27:19 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: exec(code, run_globals) 2024-09-17T00:27:19.837 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:27:19 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5579, in 2024-09-17T00:27:19.838 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:27:19 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5567, in main 2024-09-17T00:27:19.838 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:27:19 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3049, in command_deploy_from 2024-09-17T00:27:19.838 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:27:19 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3084, in _common_deploy 2024-09-17T00:27:19.838 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:27:19 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3104, in _deploy_daemon_container 2024-09-17T00:27:19.838 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:27:19 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 1124, in deploy_daemon 2024-09-17T00:27:19.838 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:27:19 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/tmp/tmpnroy8d0w.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-17T00:27:19.839 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:27:19 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: RuntimeError: Failed command: systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem: Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:27:19.839 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:27:19 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:27:19.839 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:27:19 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: 2024-09-17T00:27:19.729+0000 7f17efec7640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (6 PGs are or would become offline) 2024-09-17T00:27:19.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:27:19 smithi099 ceph-mon[56712]: pgmap v140: 161 pgs: 19 active+undersized, 13 active+undersized+degraded, 129 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 45/726 objects degraded (6.198%) 2024-09-17T00:27:19.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:27:19 smithi099 ceph-mon[56712]: Health check update: Degraded data redundancy: 45/726 objects degraded (6.198%), 13 pgs degraded (PG_DEGRADED) 2024-09-17T00:27:20.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:27:20 smithi099 ceph-mon[56712]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi002.vmouem ... 2024-09-17T00:27:20.967 INFO:journalctl@ceph.mon.b.smithi099.stdout: Creating ceph-iscsi config... 2024-09-17T00:27:20.967 INFO:journalctl@ceph.mon.b.smithi099.stdout: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/iscsi-gateway.cfg 2024-09-17T00:27:20.967 INFO:journalctl@ceph.mon.b.smithi099.stdout: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/tcmu-runner-entrypoint.sh 2024-09-17T00:27:20.968 INFO:journalctl@ceph.mon.b.smithi099.stdout: Non-zero exit code 1 from systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem 2024-09-17T00:27:20.968 INFO:journalctl@ceph.mon.b.smithi099.stdout: systemctl: stderr Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:27:20.968 INFO:journalctl@ceph.mon.b.smithi099.stdout: systemctl: stderr See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:27:20.968 INFO:journalctl@ceph.mon.b.smithi099.stdout: Traceback (most recent call last): 2024-09-17T00:27:20.968 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-17T00:27:20.968 INFO:journalctl@ceph.mon.b.smithi099.stdout: return _run_code(code, main_globals, None, 2024-09-17T00:27:20.968 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-17T00:27:20.968 INFO:journalctl@ceph.mon.b.smithi099.stdout: exec(code, run_globals) 2024-09-17T00:27:20.968 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5579, in 2024-09-17T00:27:20.968 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5567, in main 2024-09-17T00:27:20.968 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3049, in command_deploy_from 2024-09-17T00:27:20.968 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3084, in _common_deploy 2024-09-17T00:27:20.968 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3104, in _deploy_daemon_container 2024-09-17T00:27:20.969 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 1124, in deploy_daemon 2024-09-17T00:27:20.969 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-17T00:27:20.969 INFO:journalctl@ceph.mon.b.smithi099.stdout: RuntimeError: Failed command: systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem: Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:27:20.969 INFO:journalctl@ceph.mon.b.smithi099.stdout: See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:27:20.969 INFO:journalctl@ceph.mon.b.smithi099.stdout: Traceback (most recent call last): 2024-09-17T00:27:20.969 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1134, in _check_daemons 2024-09-17T00:27:20.969 INFO:journalctl@ceph.mon.b.smithi099.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-09-17T00:27:20.969 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2473, in _daemon_action 2024-09-17T00:27:20.969 INFO:journalctl@ceph.mon.b.smithi099.stdout: return self.wait_async( 2024-09-17T00:27:20.969 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 796, in wait_async 2024-09-17T00:27:20.969 INFO:journalctl@ceph.mon.b.smithi099.stdout: return self.event_loop.get_result(coro, timeout) 2024-09-17T00:27:20.969 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-09-17T00:27:20.969 INFO:journalctl@ceph.mon.b.smithi099.stdout: return future.result(timeout) 2024-09-17T00:27:20.970 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/lib64/python3.9/concurrent/futures/_base.py", line 446, in result 2024-09-17T00:27:20.970 INFO:journalctl@ceph.mon.b.smithi099.stdout: return self.__get_result() 2024-09-17T00:27:20.970 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/lib64/python3.9/concurrent/futures/_base.py", line 391, in __get_result 2024-09-17T00:27:20.970 INFO:journalctl@ceph.mon.b.smithi099.stdout: raise self._exception 2024-09-17T00:27:20.970 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1370, in _create_daemon 2024-09-17T00:27:20.970 INFO:journalctl@ceph.mon.b.smithi099.stdout: out, err, code = await self._run_cephadm( 2024-09-17T00:27:20.970 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1713, in _run_cephadm 2024-09-17T00:27:20.970 INFO:journalctl@ceph.mon.b.smithi099.stdout: raise OrchestratorError( 2024-09-17T00:27:20.970 INFO:journalctl@ceph.mon.b.smithi099.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi002.vmouem ... 2024-09-17T00:27:20.970 INFO:journalctl@ceph.mon.b.smithi099.stdout: Creating ceph-iscsi config... 2024-09-17T00:27:20.970 INFO:journalctl@ceph.mon.b.smithi099.stdout: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/iscsi-gateway.cfg 2024-09-17T00:27:20.970 INFO:journalctl@ceph.mon.b.smithi099.stdout: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/tcmu-runner-entrypoint.sh 2024-09-17T00:27:20.970 INFO:journalctl@ceph.mon.b.smithi099.stdout: Non-zero exit code 1 from systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem 2024-09-17T00:27:20.970 INFO:journalctl@ceph.mon.b.smithi099.stdout: systemctl: stderr Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:27:20.971 INFO:journalctl@ceph.mon.b.smithi099.stdout: systemctl: stderr See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:27:20.971 INFO:journalctl@ceph.mon.b.smithi099.stdout: Traceback (most recent call last): 2024-09-17T00:27:20.971 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-17T00:27:20.971 INFO:journalctl@ceph.mon.b.smithi099.stdout: return _run_code(code, main_globals, None, 2024-09-17T00:27:20.971 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-17T00:27:20.971 INFO:journalctl@ceph.mon.b.smithi099.stdout: exec(code, run_globals) 2024-09-17T00:27:20.971 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5579, in 2024-09-17T00:27:20.971 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5567, in main 2024-09-17T00:27:20.971 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3049, in command_deploy_from 2024-09-17T00:27:20.971 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3084, in _common_deploy 2024-09-17T00:27:20.971 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3104, in _deploy_daemon_container 2024-09-17T00:27:20.971 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 1124, in deploy_daemon 2024-09-17T00:27:20.971 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-17T00:27:20.972 INFO:journalctl@ceph.mon.b.smithi099.stdout: RuntimeError: Failed command: systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem: Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:27:20.972 INFO:journalctl@ceph.mon.b.smithi099.stdout: See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:27:20.972 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:27:20 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T00:27:20.972 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:27:20 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:27:20.972 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:27:20 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:27:20.972 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:27:20 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:27:20.972 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:27:20 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-09-17T00:27:20.972 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:27:20 smithi099 ceph-mon[56712]: from='mon.1 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-09-17T00:27:20.972 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:27:20 smithi099 ceph-mon[56712]: Upgrade: unsafe to stop osd(s) at this time (6 PGs are or would become offline) 2024-09-17T00:27:20.972 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:27:20 smithi099 ceph-mon[56712]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:27:20.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:27:20 smithi002 ceph-mon[66732]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi002.vmouem ... 2024-09-17T00:27:20.979 INFO:journalctl@ceph.mon.c.smithi002.stdout: Creating ceph-iscsi config... 2024-09-17T00:27:20.979 INFO:journalctl@ceph.mon.c.smithi002.stdout: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/iscsi-gateway.cfg 2024-09-17T00:27:20.979 INFO:journalctl@ceph.mon.c.smithi002.stdout: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/tcmu-runner-entrypoint.sh 2024-09-17T00:27:20.979 INFO:journalctl@ceph.mon.c.smithi002.stdout: Non-zero exit code 1 from systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem 2024-09-17T00:27:20.979 INFO:journalctl@ceph.mon.c.smithi002.stdout: systemctl: stderr Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:27:20.980 INFO:journalctl@ceph.mon.c.smithi002.stdout: systemctl: stderr See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:27:20.980 INFO:journalctl@ceph.mon.c.smithi002.stdout: Traceback (most recent call last): 2024-09-17T00:27:20.980 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-17T00:27:20.980 INFO:journalctl@ceph.mon.c.smithi002.stdout: return _run_code(code, main_globals, None, 2024-09-17T00:27:20.980 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-17T00:27:20.980 INFO:journalctl@ceph.mon.c.smithi002.stdout: exec(code, run_globals) 2024-09-17T00:27:20.980 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5579, in 2024-09-17T00:27:20.980 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5567, in main 2024-09-17T00:27:20.980 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3049, in command_deploy_from 2024-09-17T00:27:20.980 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3084, in _common_deploy 2024-09-17T00:27:20.980 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3104, in _deploy_daemon_container 2024-09-17T00:27:20.980 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 1124, in deploy_daemon 2024-09-17T00:27:20.980 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-17T00:27:20.981 INFO:journalctl@ceph.mon.c.smithi002.stdout: RuntimeError: Failed command: systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem: Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:27:20.981 INFO:journalctl@ceph.mon.c.smithi002.stdout: See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:27:20.981 INFO:journalctl@ceph.mon.c.smithi002.stdout: Traceback (most recent call last): 2024-09-17T00:27:20.981 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1134, in _check_daemons 2024-09-17T00:27:20.981 INFO:journalctl@ceph.mon.c.smithi002.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-09-17T00:27:20.981 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2473, in _daemon_action 2024-09-17T00:27:20.981 INFO:journalctl@ceph.mon.c.smithi002.stdout: return self.wait_async( 2024-09-17T00:27:20.981 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 796, in wait_async 2024-09-17T00:27:20.981 INFO:journalctl@ceph.mon.c.smithi002.stdout: return self.event_loop.get_result(coro, timeout) 2024-09-17T00:27:20.981 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-09-17T00:27:20.981 INFO:journalctl@ceph.mon.c.smithi002.stdout: return future.result(timeout) 2024-09-17T00:27:20.981 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/lib64/python3.9/concurrent/futures/_base.py", line 446, in result 2024-09-17T00:27:20.982 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:27:20 smithi002 ceph-mon[63804]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi002.vmouem ... 2024-09-17T00:27:20.982 INFO:journalctl@ceph.mon.a.smithi002.stdout: Creating ceph-iscsi config... 2024-09-17T00:27:20.982 INFO:journalctl@ceph.mon.a.smithi002.stdout: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/iscsi-gateway.cfg 2024-09-17T00:27:20.982 INFO:journalctl@ceph.mon.a.smithi002.stdout: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/tcmu-runner-entrypoint.sh 2024-09-17T00:27:20.982 INFO:journalctl@ceph.mon.a.smithi002.stdout: Non-zero exit code 1 from systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem 2024-09-17T00:27:20.982 INFO:journalctl@ceph.mon.a.smithi002.stdout: systemctl: stderr Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:27:20.982 INFO:journalctl@ceph.mon.a.smithi002.stdout: systemctl: stderr See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:27:20.982 INFO:journalctl@ceph.mon.a.smithi002.stdout: Traceback (most recent call last): 2024-09-17T00:27:20.983 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-17T00:27:20.983 INFO:journalctl@ceph.mon.a.smithi002.stdout: return _run_code(code, main_globals, None, 2024-09-17T00:27:20.983 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-17T00:27:20.983 INFO:journalctl@ceph.mon.a.smithi002.stdout: exec(code, run_globals) 2024-09-17T00:27:20.983 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5579, in 2024-09-17T00:27:20.983 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5567, in main 2024-09-17T00:27:20.983 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3049, in command_deploy_from 2024-09-17T00:27:20.983 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3084, in _common_deploy 2024-09-17T00:27:20.983 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3104, in _deploy_daemon_container 2024-09-17T00:27:20.983 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 1124, in deploy_daemon 2024-09-17T00:27:20.983 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-17T00:27:20.984 INFO:journalctl@ceph.mon.a.smithi002.stdout: RuntimeError: Failed command: systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem: Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:27:20.984 INFO:journalctl@ceph.mon.a.smithi002.stdout: See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:27:20.984 INFO:journalctl@ceph.mon.a.smithi002.stdout: Traceback (most recent call last): 2024-09-17T00:27:20.984 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1134, in _check_daemons 2024-09-17T00:27:20.984 INFO:journalctl@ceph.mon.a.smithi002.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-09-17T00:27:20.984 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2473, in _daemon_action 2024-09-17T00:27:20.984 INFO:journalctl@ceph.mon.a.smithi002.stdout: return self.wait_async( 2024-09-17T00:27:20.984 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 796, in wait_async 2024-09-17T00:27:20.984 INFO:journalctl@ceph.mon.a.smithi002.stdout: return self.event_loop.get_result(coro, timeout) 2024-09-17T00:27:20.984 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-09-17T00:27:20.984 INFO:journalctl@ceph.mon.a.smithi002.stdout: return future.result(timeout) 2024-09-17T00:27:20.984 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/lib64/python3.9/concurrent/futures/_base.py", line 446, in result 2024-09-17T00:27:20.984 INFO:journalctl@ceph.mon.a.smithi002.stdout: return self.__get_result() 2024-09-17T00:27:20.984 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/lib64/python3.9/concurrent/futures/_base.py", line 391, in __get_result 2024-09-17T00:27:20.985 INFO:journalctl@ceph.mon.a.smithi002.stdout: raise self._exception 2024-09-17T00:27:20.985 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1370, in _create_daemon 2024-09-17T00:27:20.985 INFO:journalctl@ceph.mon.a.smithi002.stdout: out, err, code = await self._run_cephadm( 2024-09-17T00:27:20.985 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1713, in _run_cephadm 2024-09-17T00:27:20.985 INFO:journalctl@ceph.mon.a.smithi002.stdout: raise OrchestratorError( 2024-09-17T00:27:20.985 INFO:journalctl@ceph.mon.a.smithi002.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi002.vmouem ... 2024-09-17T00:27:20.985 INFO:journalctl@ceph.mon.a.smithi002.stdout: Creating ceph-iscsi config... 2024-09-17T00:27:20.985 INFO:journalctl@ceph.mon.a.smithi002.stdout: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/iscsi-gateway.cfg 2024-09-17T00:27:20.985 INFO:journalctl@ceph.mon.a.smithi002.stdout: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/tcmu-runner-entrypoint.sh 2024-09-17T00:27:20.985 INFO:journalctl@ceph.mon.a.smithi002.stdout: Non-zero exit code 1 from systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem 2024-09-17T00:27:20.985 INFO:journalctl@ceph.mon.a.smithi002.stdout: systemctl: stderr Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:27:20.985 INFO:journalctl@ceph.mon.a.smithi002.stdout: systemctl: stderr See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:27:20.985 INFO:journalctl@ceph.mon.a.smithi002.stdout: Traceback (most recent call last): 2024-09-17T00:27:20.985 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-17T00:27:20.986 INFO:journalctl@ceph.mon.a.smithi002.stdout: return _run_code(code, main_globals, None, 2024-09-17T00:27:20.986 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-17T00:27:20.986 INFO:journalctl@ceph.mon.a.smithi002.stdout: exec(code, run_globals) 2024-09-17T00:27:20.986 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5579, in 2024-09-17T00:27:20.986 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5567, in main 2024-09-17T00:27:20.986 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3049, in command_deploy_from 2024-09-17T00:27:20.986 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3084, in _common_deploy 2024-09-17T00:27:20.986 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3104, in _deploy_daemon_container 2024-09-17T00:27:20.986 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 1124, in deploy_daemon 2024-09-17T00:27:20.986 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-17T00:27:20.986 INFO:journalctl@ceph.mon.a.smithi002.stdout: RuntimeError: Failed command: systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem: Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:27:20.986 INFO:journalctl@ceph.mon.a.smithi002.stdout: See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:27:20.986 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:27:20 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T00:27:20.986 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:27:20 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:27:20.987 INFO:journalctl@ceph.mon.c.smithi002.stdout: return self.__get_result() 2024-09-17T00:27:20.987 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/lib64/python3.9/concurrent/futures/_base.py", line 391, in __get_result 2024-09-17T00:27:20.987 INFO:journalctl@ceph.mon.c.smithi002.stdout: raise self._exception 2024-09-17T00:27:20.987 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1370, in _create_daemon 2024-09-17T00:27:20.987 INFO:journalctl@ceph.mon.c.smithi002.stdout: out, err, code = await self._run_cephadm( 2024-09-17T00:27:20.987 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1713, in _run_cephadm 2024-09-17T00:27:20.987 INFO:journalctl@ceph.mon.c.smithi002.stdout: raise OrchestratorError( 2024-09-17T00:27:20.987 INFO:journalctl@ceph.mon.c.smithi002.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi002.vmouem ... 2024-09-17T00:27:20.987 INFO:journalctl@ceph.mon.c.smithi002.stdout: Creating ceph-iscsi config... 2024-09-17T00:27:20.988 INFO:journalctl@ceph.mon.c.smithi002.stdout: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/iscsi-gateway.cfg 2024-09-17T00:27:20.988 INFO:journalctl@ceph.mon.c.smithi002.stdout: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/tcmu-runner-entrypoint.sh 2024-09-17T00:27:20.988 INFO:journalctl@ceph.mon.c.smithi002.stdout: Non-zero exit code 1 from systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem 2024-09-17T00:27:20.988 INFO:journalctl@ceph.mon.c.smithi002.stdout: systemctl: stderr Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:27:20.988 INFO:journalctl@ceph.mon.c.smithi002.stdout: systemctl: stderr See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:27:20.988 INFO:journalctl@ceph.mon.c.smithi002.stdout: Traceback (most recent call last): 2024-09-17T00:27:20.988 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-17T00:27:20.988 INFO:journalctl@ceph.mon.c.smithi002.stdout: return _run_code(code, main_globals, None, 2024-09-17T00:27:20.988 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-17T00:27:20.988 INFO:journalctl@ceph.mon.c.smithi002.stdout: exec(code, run_globals) 2024-09-17T00:27:20.988 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5579, in 2024-09-17T00:27:20.988 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5567, in main 2024-09-17T00:27:20.988 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3049, in command_deploy_from 2024-09-17T00:27:20.989 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3084, in _common_deploy 2024-09-17T00:27:20.989 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3104, in _deploy_daemon_container 2024-09-17T00:27:20.989 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 1124, in deploy_daemon 2024-09-17T00:27:20.989 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-17T00:27:20.989 INFO:journalctl@ceph.mon.c.smithi002.stdout: RuntimeError: Failed command: systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem: Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:27:20.989 INFO:journalctl@ceph.mon.c.smithi002.stdout: See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:27:20.989 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:27:20 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T00:27:20.989 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:27:20 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:27:20.989 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:27:20 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:27:20.989 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:27:20 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:27:20.989 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:27:20 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-09-17T00:27:20.989 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:27:20 smithi002 ceph-mon[66732]: from='mon.1 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-09-17T00:27:20.989 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:27:20 smithi002 ceph-mon[66732]: Upgrade: unsafe to stop osd(s) at this time (6 PGs are or would become offline) 2024-09-17T00:27:20.989 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:27:20 smithi002 ceph-mon[66732]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:27:20.990 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:27:20 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:27:20.990 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:27:20 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:27:20.990 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:27:20 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-09-17T00:27:20.990 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:27:20 smithi002 ceph-mon[63804]: from='mon.1 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-09-17T00:27:20.990 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:27:20 smithi002 ceph-mon[63804]: Upgrade: unsafe to stop osd(s) at this time (6 PGs are or would become offline) 2024-09-17T00:27:20.990 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:27:20 smithi002 ceph-mon[63804]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:27:21.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:27:21 smithi099 ceph-mon[56712]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 45/726 objects degraded (6.198%), 13 pgs degraded) 2024-09-17T00:27:21.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:27:21 smithi099 ceph-mon[56712]: Cluster is now healthy 2024-09-17T00:27:21.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:27:21 smithi002 ceph-mon[63804]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 45/726 objects degraded (6.198%), 13 pgs degraded) 2024-09-17T00:27:21.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:27:21 smithi002 ceph-mon[63804]: Cluster is now healthy 2024-09-17T00:27:21.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:27:21 smithi002 ceph-mon[66732]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 45/726 objects degraded (6.198%), 13 pgs degraded) 2024-09-17T00:27:21.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:27:21 smithi002 ceph-mon[66732]: Cluster is now healthy 2024-09-17T00:27:22.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:27:22 smithi099 ceph-mon[56712]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:27:22.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:27:22 smithi002 ceph-mon[63804]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:27:22.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:27:22 smithi002 ceph-mon[66732]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:27:25.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:27:25 smithi002 ceph-mon[63804]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:27:25.480 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:27:25 smithi002 ceph-mon[66732]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:27:25.670 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:27:25 smithi099 ceph-mon[56712]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:27:26.979 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:27:26 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: ::ffff:172.21.15.99 - - [17/Sep/2024:00:27:26] "GET /metrics HTTP/1.1" 200 38438 "" "Prometheus/2.43.0" 2024-09-17T00:27:27.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:27:27 smithi099 ceph-mon[56712]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:27:27.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:27:27 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:27:27.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:27:27 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T00:27:27.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:27:27 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:27:27.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:27:27 smithi002 ceph-mon[63804]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:27:27.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:27:27 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:27:27.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:27:27 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T00:27:27.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:27:27 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:27:27.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:27:27 smithi002 ceph-mon[66732]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:27:27.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:27:27 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:27:27.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:27:27 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T00:27:27.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:27:27 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:27:28.993 INFO:teuthology.orchestra.run.smithi002.stdout:true 2024-09-17T00:27:29.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:27:29 smithi002 ceph-mon[66732]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:27:29.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:27:29 smithi002 ceph-mon[63804]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:27:29.677 INFO:teuthology.orchestra.run.smithi002.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-17T00:27:29.677 INFO:teuthology.orchestra.run.smithi002.stdout:alertmanager.a smithi002 *:9093,9094 running (5m) 17s ago 10m 26.1M - 0.25.0 c8568f914cd2 a21e13b3d677 2024-09-17T00:27:29.677 INFO:teuthology.orchestra.run.smithi002.stdout:grafana.a smithi099 *:3000 running (4m) 3m ago 10m 46.9M - 8.3.5 dad864ee21e9 bafb824a3c08 2024-09-17T00:27:29.677 INFO:teuthology.orchestra.run.smithi002.stdout:iscsi.foo.smithi002.vmouem smithi002 running (37s) 17s ago 9m 48.3M - 3.5 e1d6a67b021e b5424a0496a5 2024-09-17T00:27:29.677 INFO:teuthology.orchestra.run.smithi002.stdout:mgr.x smithi099 *:8443,9283,8765 running (4m) 3m ago 14m 439M - 19.1.1-330-gc2ddf972 ebb0615149de 6aa2ccef9ca0 2024-09-17T00:27:29.677 INFO:teuthology.orchestra.run.smithi002.stdout:mgr.y smithi002 *:8443,9283,8765 running (4m) 17s ago 16m 506M - 19.1.1-330-gc2ddf972 ebb0615149de 8838897e8449 2024-09-17T00:27:29.677 INFO:teuthology.orchestra.run.smithi002.stdout:mon.a smithi002 running (3m) 17s ago 16m 51.3M 2048M 19.1.1-330-gc2ddf972 ebb0615149de a035def53ef6 2024-09-17T00:27:29.677 INFO:teuthology.orchestra.run.smithi002.stdout:mon.b smithi099 running (3m) 3m ago 15m 17.5M 2048M 19.1.1-330-gc2ddf972 ebb0615149de aab3ab73a850 2024-09-17T00:27:29.677 INFO:teuthology.orchestra.run.smithi002.stdout:mon.c smithi002 running (3m) 17s ago 15m 43.4M 2048M 19.1.1-330-gc2ddf972 ebb0615149de 8a510255ae66 2024-09-17T00:27:29.678 INFO:teuthology.orchestra.run.smithi002.stdout:node-exporter.a smithi002 *:9100 running (5m) 17s ago 10m 14.8M - 1.5.0 0da6a335fe13 1030ae01bb32 2024-09-17T00:27:29.678 INFO:teuthology.orchestra.run.smithi002.stdout:node-exporter.b smithi099 *:9100 running (4m) 3m ago 10m 16.8M - 1.5.0 0da6a335fe13 01fcd35dca8a 2024-09-17T00:27:29.678 INFO:teuthology.orchestra.run.smithi002.stdout:osd.0 smithi002 running (2m) 17s ago 14m 45.8M 2737M 19.1.1-330-gc2ddf972 ebb0615149de 8345719cb6f0 2024-09-17T00:27:29.678 INFO:teuthology.orchestra.run.smithi002.stdout:osd.1 smithi002 running (99s) 17s ago 14m 51.5M 2737M 19.1.1-330-gc2ddf972 ebb0615149de 3dbc23d57c53 2024-09-17T00:27:29.678 INFO:teuthology.orchestra.run.smithi002.stdout:osd.2 smithi002 running (60s) 17s ago 13m 49.3M 2737M 19.1.1-330-gc2ddf972 ebb0615149de d21401b6763b 2024-09-17T00:27:29.678 INFO:teuthology.orchestra.run.smithi002.stdout:osd.3 smithi002 running (22s) 17s ago 13m 12.2M 2737M 19.1.1-330-gc2ddf972 ebb0615149de 980f2a25d096 2024-09-17T00:27:29.678 INFO:teuthology.orchestra.run.smithi002.stdout:osd.4 smithi099 running (12m) 3m ago 12m 55.4M 2992M 17.2.0 e1d6a67b021e 0af1827b658d 2024-09-17T00:27:29.678 INFO:teuthology.orchestra.run.smithi002.stdout:osd.5 smithi099 running (12m) 3m ago 12m 54.2M 2992M 17.2.0 e1d6a67b021e 4f7ef3aefc58 2024-09-17T00:27:29.678 INFO:teuthology.orchestra.run.smithi002.stdout:osd.6 smithi099 running (12m) 3m ago 12m 51.0M 2992M 17.2.0 e1d6a67b021e 3213f5e6952a 2024-09-17T00:27:29.678 INFO:teuthology.orchestra.run.smithi002.stdout:osd.7 smithi099 running (11m) 3m ago 11m 56.6M 2992M 17.2.0 e1d6a67b021e 63be3a1ed0a1 2024-09-17T00:27:29.678 INFO:teuthology.orchestra.run.smithi002.stdout:prometheus.a smithi099 *:9095 running (4m) 3m ago 11m 50.1M - 2.43.0 a07b618ecd1d bf32c2ae11dc 2024-09-17T00:27:29.678 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.foo.smithi002.uwdhcs smithi002 *:8000 running (9m) 17s ago 9m 76.1M - 17.2.0 e1d6a67b021e 9c35813f459a 2024-09-17T00:27:29.678 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.foo.smithi099.utzfap smithi099 *:8000 running (9m) 3m ago 9m 76.0M - 17.2.0 e1d6a67b021e 34ec989160a6 2024-09-17T00:27:29.679 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.smpl.smithi002.rqnxii smithi002 *:80 running (9m) 17s ago 9m 74.4M - 17.2.0 e1d6a67b021e 0e180d0afb99 2024-09-17T00:27:29.679 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.smpl.smithi099.nzvkpm smithi099 *:80 running (9m) 3m ago 9m 76.5M - 17.2.0 e1d6a67b021e 6a068e82ac72 2024-09-17T00:27:29.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:27:29 smithi099 ceph-mon[56712]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:27:30.120 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-09-17T00:27:30.120 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": { 2024-09-17T00:27:30.120 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 19.1.1-330-gc2ddf972 (c2ddf9721ef254645201cdbabd9c8f49c30a586f) squid (stable)": 3 2024-09-17T00:27:30.120 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-09-17T00:27:30.120 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": { 2024-09-17T00:27:30.120 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 19.1.1-330-gc2ddf972 (c2ddf9721ef254645201cdbabd9c8f49c30a586f) squid (stable)": 2 2024-09-17T00:27:30.120 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-09-17T00:27:30.120 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": { 2024-09-17T00:27:30.120 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4, 2024-09-17T00:27:30.121 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 19.1.1-330-gc2ddf972 (c2ddf9721ef254645201cdbabd9c8f49c30a586f) squid (stable)": 4 2024-09-17T00:27:30.121 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-09-17T00:27:30.121 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw": { 2024-09-17T00:27:30.121 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-09-17T00:27:30.121 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-09-17T00:27:30.121 INFO:teuthology.orchestra.run.smithi002.stdout: "overall": { 2024-09-17T00:27:30.121 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8, 2024-09-17T00:27:30.121 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 19.1.1-330-gc2ddf972 (c2ddf9721ef254645201cdbabd9c8f49c30a586f) squid (stable)": 9 2024-09-17T00:27:30.121 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-09-17T00:27:30.121 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-09-17T00:27:30.478 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-09-17T00:27:30.478 INFO:teuthology.orchestra.run.smithi002.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f", 2024-09-17T00:27:30.478 INFO:teuthology.orchestra.run.smithi002.stdout: "in_progress": true, 2024-09-17T00:27:30.478 INFO:teuthology.orchestra.run.smithi002.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-17T00:27:30.478 INFO:teuthology.orchestra.run.smithi002.stdout: "services_complete": [ 2024-09-17T00:27:30.478 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr", 2024-09-17T00:27:30.478 INFO:teuthology.orchestra.run.smithi002.stdout: "mon" 2024-09-17T00:27:30.479 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-09-17T00:27:30.479 INFO:teuthology.orchestra.run.smithi002.stdout: "progress": "9/23 daemons upgraded", 2024-09-17T00:27:30.479 INFO:teuthology.orchestra.run.smithi002.stdout: "message": "Currently upgrading osd daemons", 2024-09-17T00:27:30.479 INFO:teuthology.orchestra.run.smithi002.stdout: "is_paused": false 2024-09-17T00:27:30.479 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-09-17T00:27:30.534 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:27:30 smithi002 ceph-mon[66732]: from='client.44518 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:27:30.534 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:27:30 smithi002 ceph-mon[66732]: from='client.44524 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:27:30.534 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:27:30 smithi002 ceph-mon[66732]: from='client.? 172.21.15.2:0/1609535824' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:27:30.534 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:27:30 smithi002 ceph-mon[63804]: from='client.44518 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:27:30.535 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:27:30 smithi002 ceph-mon[63804]: from='client.44524 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:27:30.535 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:27:30 smithi002 ceph-mon[63804]: from='client.? 172.21.15.2:0/1609535824' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:27:30.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:27:30 smithi099 ceph-mon[56712]: from='client.44518 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:27:30.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:27:30 smithi099 ceph-mon[56712]: from='client.44524 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:27:30.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:27:30 smithi099 ceph-mon[56712]: from='client.? 172.21.15.2:0/1609535824' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:27:30.917 INFO:teuthology.orchestra.run.smithi002.stdout:HEALTH_OK 2024-09-17T00:27:31.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:27:31 smithi002 ceph-mon[63804]: from='client.34626 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:27:31.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:27:31 smithi002 ceph-mon[63804]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:27:31.376 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:27:31 smithi002 ceph-mon[63804]: from='client.34638 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:27:31.377 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:27:31 smithi002 ceph-mon[63804]: from='client.? 172.21.15.2:0/244809325' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T00:27:31.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:27:31 smithi002 ceph-mon[66732]: from='client.34626 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:27:31.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:27:31 smithi002 ceph-mon[66732]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:27:31.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:27:31 smithi002 ceph-mon[66732]: from='client.34638 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:27:31.377 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:27:31 smithi002 ceph-mon[66732]: from='client.? 172.21.15.2:0/244809325' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T00:27:31.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:27:31 smithi099 ceph-mon[56712]: from='client.34626 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:27:31.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:27:31 smithi099 ceph-mon[56712]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:27:31.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:27:31 smithi099 ceph-mon[56712]: from='client.34638 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:27:31.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:27:31 smithi099 ceph-mon[56712]: from='client.? 172.21.15.2:0/244809325' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T00:27:32.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:27:32 smithi099 ceph-mon[56712]: from='client.? 172.21.15.2:0/535038285' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-17T00:27:32.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:27:32 smithi002 ceph-mon[63804]: from='client.? 172.21.15.2:0/535038285' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-17T00:27:32.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:27:32 smithi002 ceph-mon[66732]: from='client.? 172.21.15.2:0/535038285' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-17T00:27:33.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:27:33 smithi099 ceph-mon[56712]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:27:33.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:27:33 smithi002 ceph-mon[63804]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:27:33.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:27:33 smithi002 ceph-mon[66732]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail 2024-09-17T00:27:35.420 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:27:35 smithi099 ceph-mon[56712]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-09-17T00:27:35.421 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:27:35 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-09-17T00:27:35.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:27:35 smithi002 ceph-mon[63804]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-09-17T00:27:35.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:27:35 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-09-17T00:27:35.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:27:35 smithi002 ceph-mon[66732]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-09-17T00:27:35.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:27:35 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-09-17T00:27:36.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:27:36 smithi099 ceph-mon[56712]: from='mon.1 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-09-17T00:27:36.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:27:36 smithi099 ceph-mon[56712]: Upgrade: osd.4 is safe to restart 2024-09-17T00:27:36.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:27:36 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:27:36.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:27:36 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-09-17T00:27:36.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:27:36 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:27:36.631 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:27:36 smithi002 ceph-mon[63804]: from='mon.1 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-09-17T00:27:36.631 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:27:36 smithi002 ceph-mon[63804]: Upgrade: osd.4 is safe to restart 2024-09-17T00:27:36.631 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:27:36 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:27:36.631 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:27:36 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-09-17T00:27:36.631 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:27:36 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:27:36.631 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:27:36 smithi002 ceph-mon[66732]: from='mon.1 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-09-17T00:27:36.632 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:27:36 smithi002 ceph-mon[66732]: Upgrade: osd.4 is safe to restart 2024-09-17T00:27:36.632 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:27:36 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:27:36.632 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:27:36 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-09-17T00:27:36.632 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:27:36 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:27:36.979 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:27:36 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: ::ffff:172.21.15.99 - - [17/Sep/2024:00:27:36] "GET /metrics HTTP/1.1" 200 38445 "" "Prometheus/2.43.0" 2024-09-17T00:27:37.435 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:27:37 smithi099 ceph-mon[56712]: Upgrade: Updating osd.4 2024-09-17T00:27:37.435 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:27:37 smithi099 ceph-mon[56712]: Deploying daemon osd.4 on smithi099 2024-09-17T00:27:37.435 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:27:37 smithi099 ceph-mon[56712]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-09-17T00:27:37.644 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:27:37 smithi002 ceph-mon[66732]: Upgrade: Updating osd.4 2024-09-17T00:27:37.644 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:27:37 smithi002 ceph-mon[66732]: Deploying daemon osd.4 on smithi099 2024-09-17T00:27:37.644 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:27:37 smithi002 ceph-mon[66732]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-09-17T00:27:37.644 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:27:37 smithi002 ceph-mon[63804]: Upgrade: Updating osd.4 2024-09-17T00:27:37.644 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:27:37 smithi002 ceph-mon[63804]: Deploying daemon osd.4 on smithi099 2024-09-17T00:27:37.644 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:27:37 smithi002 ceph-mon[63804]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-09-17T00:27:39.316 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:27:39 smithi099 ceph-mon[56712]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-09-17T00:27:39.316 INFO:journalctl@ceph.osd.4.smithi099.stdout:Sep 17 00:27:39 smithi099 systemd[1]: Stopping Ceph osd.4 for 3aad27a2-7489-11ef-bceb-c7b262605968... 2024-09-17T00:27:39.717 INFO:journalctl@ceph.osd.4.smithi099.stdout:Sep 17 00:27:39 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-4[27932]: 2024-09-17T00:27:39.316+0000 7f36d20ea700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-17T00:27:39.717 INFO:journalctl@ceph.osd.4.smithi099.stdout:Sep 17 00:27:39 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-4[27932]: 2024-09-17T00:27:39.316+0000 7f36d20ea700 -1 osd.4 132 *** Got signal Terminated *** 2024-09-17T00:27:39.717 INFO:journalctl@ceph.osd.4.smithi099.stdout:Sep 17 00:27:39 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-4[27932]: 2024-09-17T00:27:39.316+0000 7f36d20ea700 -1 osd.4 132 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-17T00:27:39.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:27:39 smithi002 ceph-mon[63804]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-09-17T00:27:39.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:27:39 smithi002 ceph-mon[66732]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-09-17T00:27:40.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:27:40 smithi099 ceph-mon[56712]: osd.4 marked itself down and dead 2024-09-17T00:27:40.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:27:40 smithi002 ceph-mon[63804]: osd.4 marked itself down and dead 2024-09-17T00:27:40.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:27:40 smithi002 ceph-mon[66732]: osd.4 marked itself down and dead 2024-09-17T00:27:40.999 INFO:journalctl@ceph.osd.4.smithi099.stdout:Sep 17 00:27:40 smithi099 podman[59490]: 2024-09-17 00:27:40.749271003 +0000 UTC m=+1.559488401 container died 0af1827b658df1b845619b2f7088cbf6e7da88d13194c55ccd46965dee93c427 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-4, GIT_CLEAN=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.buildah.version=1.19.8, maintainer=Guillaume Abrioux , version=8, GIT_BRANCH=HEAD, RELEASE=HEAD, vcs-type=git, ceph=True, io.openshift.expose-services=, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, name=centos-stream, CEPH_POINT_RELEASE=-17.2.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-05-03T08:36:31.336870, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.tags=base centos centos-stream, architecture=x86_64, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.display-name=CentOS Stream 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.component=centos-stream-container, release=754, distribution-scope=public, vendor=Red Hat, Inc.) 2024-09-17T00:27:41.263 INFO:journalctl@ceph.osd.4.smithi099.stdout:Sep 17 00:27:41 smithi099 podman[59490]: 2024-09-17 00:27:41.101601508 +0000 UTC m=+1.911818886 container cleanup 0af1827b658df1b845619b2f7088cbf6e7da88d13194c55ccd46965dee93c427 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-4, io.openshift.expose-services=, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_BRANCH=HEAD, distribution-scope=public, RELEASE=HEAD, ceph=True, name=centos-stream, version=8, CEPH_POINT_RELEASE=-17.2.0, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, architecture=x86_64, build-date=2022-05-03T08:36:31.336870, maintainer=Guillaume Abrioux , vendor=Red Hat, Inc., GIT_CLEAN=True, io.k8s.display-name=CentOS Stream 8, vcs-type=git, release=754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.buildah.version=1.19.8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base centos centos-stream, com.redhat.component=centos-stream-container, GIT_REPO=https://github.com/ceph/ceph-container.git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream) 2024-09-17T00:27:41.263 INFO:journalctl@ceph.osd.4.smithi099.stdout:Sep 17 00:27:41 smithi099 bash[59490]: ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-4 2024-09-17T00:27:41.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:27:41 smithi099 ceph-mon[56712]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-09-17T00:27:41.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:27:41 smithi099 ceph-mon[56712]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-17T00:27:41.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:27:41 smithi099 ceph-mon[56712]: osdmap e133: 8 total, 7 up, 8 in 2024-09-17T00:27:41.717 INFO:journalctl@ceph.osd.4.smithi099.stdout:Sep 17 00:27:41 smithi099 podman[59507]: 2024-09-17 00:27:41.411686831 +0000 UTC m=+0.660486339 container remove 0af1827b658df1b845619b2f7088cbf6e7da88d13194c55ccd46965dee93c427 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-4, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.display-name=CentOS Stream 8, architecture=x86_64, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_CLEAN=True, CEPH_POINT_RELEASE=-17.2.0, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.expose-services=, distribution-scope=public, com.redhat.component=centos-stream-container, maintainer=Guillaume Abrioux , summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, build-date=2022-05-03T08:36:31.336870, version=8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, RELEASE=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, name=centos-stream, GIT_BRANCH=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, ceph=True, io.buildah.version=1.19.8, release=754, vendor=Red Hat, Inc., io.openshift.tags=base centos centos-stream, vcs-type=git) 2024-09-17T00:27:41.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:27:41 smithi002 ceph-mon[63804]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-09-17T00:27:41.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:27:41 smithi002 ceph-mon[63804]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-17T00:27:41.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:27:41 smithi002 ceph-mon[63804]: osdmap e133: 8 total, 7 up, 8 in 2024-09-17T00:27:41.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:27:41 smithi002 ceph-mon[66732]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-09-17T00:27:41.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:27:41 smithi002 ceph-mon[66732]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-17T00:27:41.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:27:41 smithi002 ceph-mon[66732]: osdmap e133: 8 total, 7 up, 8 in 2024-09-17T00:27:42.103 INFO:journalctl@ceph.osd.4.smithi099.stdout:Sep 17 00:27:41 smithi099 podman[59592]: 2024-09-17 00:27:41.670646973 +0000 UTC m=+0.015866530 image pull ebb0615149def4a3179c82c0cc47af75808300abb9f1142788e5e605a7ac0e5b quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f 2024-09-17T00:27:42.104 INFO:journalctl@ceph.osd.4.smithi099.stdout:Sep 17 00:27:41 smithi099 podman[59592]: 2024-09-17 00:27:41.779194444 +0000 UTC m=+0.124414048 container create 0aca1dfe3ad11a2b6445315425cb65ed536a07cbd40c8250af1bf064d3348953 (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-4-deactivate, org.label-schema.build-date=20240731, RELEASE=squid-c2ddf97, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-09-17T00:27:42.104 INFO:journalctl@ceph.osd.4.smithi099.stdout:Sep 17 00:27:42 smithi099 podman[59592]: 2024-09-17 00:27:42.006654071 +0000 UTC m=+0.351873638 container init 0aca1dfe3ad11a2b6445315425cb65ed536a07cbd40c8250af1bf064d3348953 (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-4-deactivate, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-c2ddf97) 2024-09-17T00:27:42.104 INFO:journalctl@ceph.osd.4.smithi099.stdout:Sep 17 00:27:42 smithi099 podman[59592]: 2024-09-17 00:27:42.009927514 +0000 UTC m=+0.355147075 container start 0aca1dfe3ad11a2b6445315425cb65ed536a07cbd40c8250af1bf064d3348953 (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-4-deactivate, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, RELEASE=squid-c2ddf97, maintainer=Guillaume Abrioux ) 2024-09-17T00:27:42.104 INFO:journalctl@ceph.osd.4.smithi099.stdout:Sep 17 00:27:42 smithi099 podman[59592]: 2024-09-17 00:27:42.106759447 +0000 UTC m=+0.451979008 container attach 0aca1dfe3ad11a2b6445315425cb65ed536a07cbd40c8250af1bf064d3348953 (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-4-deactivate, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-c2ddf97, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True) 2024-09-17T00:27:42.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:27:42 smithi099 ceph-mon[56712]: osdmap e134: 8 total, 7 up, 8 in 2024-09-17T00:27:42.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:27:42 smithi099 ceph-mon[56712]: from='client.44563 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:27:42.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:27:42 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:27:42.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:27:42 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T00:27:42.468 INFO:journalctl@ceph.osd.4.smithi099.stdout:Sep 17 00:27:42 smithi099 podman[59592]: 2024-09-17 00:27:42.203833955 +0000 UTC m=+0.549053524 container died 0aca1dfe3ad11a2b6445315425cb65ed536a07cbd40c8250af1bf064d3348953 (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-4-deactivate, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=squid-c2ddf97, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-09-17T00:27:42.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:27:42 smithi002 ceph-mon[63804]: osdmap e134: 8 total, 7 up, 8 in 2024-09-17T00:27:42.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:27:42 smithi002 ceph-mon[63804]: from='client.44563 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:27:42.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:27:42 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:27:42.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:27:42 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T00:27:42.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:27:42 smithi002 ceph-mon[66732]: osdmap e134: 8 total, 7 up, 8 in 2024-09-17T00:27:42.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:27:42 smithi002 ceph-mon[66732]: from='client.44563 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:27:42.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:27:42 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:27:42.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:27:42 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T00:27:43.187 INFO:journalctl@ceph.osd.4.smithi099.stdout:Sep 17 00:27:42 smithi099 podman[59615]: 2024-09-17 00:27:42.760648243 +0000 UTC m=+0.551247773 container remove 0aca1dfe3ad11a2b6445315425cb65ed536a07cbd40c8250af1bf064d3348953 (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-4-deactivate, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, RELEASE=squid-c2ddf97, io.buildah.version=1.37.2, org.label-schema.license=GPLv2) 2024-09-17T00:27:43.188 INFO:journalctl@ceph.osd.4.smithi099.stdout:Sep 17 00:27:42 smithi099 systemd[1]: ceph-3aad27a2-7489-11ef-bceb-c7b262605968@osd.4.service: Deactivated successfully. 2024-09-17T00:27:43.188 INFO:journalctl@ceph.osd.4.smithi099.stdout:Sep 17 00:27:42 smithi099 systemd[1]: Stopped Ceph osd.4 for 3aad27a2-7489-11ef-bceb-c7b262605968. 2024-09-17T00:27:43.188 INFO:journalctl@ceph.osd.4.smithi099.stdout:Sep 17 00:27:42 smithi099 systemd[1]: ceph-3aad27a2-7489-11ef-bceb-c7b262605968@osd.4.service: Consumed 9.487s CPU time. 2024-09-17T00:27:43.188 INFO:journalctl@ceph.osd.4.smithi099.stdout:Sep 17 00:27:42 smithi099 systemd[1]: Starting Ceph osd.4 for 3aad27a2-7489-11ef-bceb-c7b262605968... 2024-09-17T00:27:43.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:27:43 smithi099 ceph-mon[56712]: pgmap v154: 161 pgs: 7 active+undersized, 9 peering, 18 stale+active+clean, 4 active+undersized+degraded, 123 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 19/726 objects degraded (2.617%) 2024-09-17T00:27:43.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:27:43 smithi099 ceph-mon[56712]: Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY) 2024-09-17T00:27:43.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:27:43 smithi099 ceph-mon[56712]: Health check failed: Degraded data redundancy: 19/726 objects degraded (2.617%), 4 pgs degraded (PG_DEGRADED) 2024-09-17T00:27:43.467 INFO:journalctl@ceph.osd.4.smithi099.stdout:Sep 17 00:27:43 smithi099 podman[59714]: 2024-09-17 00:27:43.093537332 +0000 UTC m=+0.015918314 image pull ebb0615149def4a3179c82c0cc47af75808300abb9f1142788e5e605a7ac0e5b quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f 2024-09-17T00:27:43.467 INFO:journalctl@ceph.osd.4.smithi099.stdout:Sep 17 00:27:43 smithi099 podman[59714]: 2024-09-17 00:27:43.212951557 +0000 UTC m=+0.135332574 container create e73cc5df988f3d90794083afb65371aa84db58f481d90fec71ae809136875237 (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-4-activate, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=squid-c2ddf97, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS) 2024-09-17T00:27:43.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:27:43 smithi002 ceph-mon[63804]: pgmap v154: 161 pgs: 7 active+undersized, 9 peering, 18 stale+active+clean, 4 active+undersized+degraded, 123 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 19/726 objects degraded (2.617%) 2024-09-17T00:27:43.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:27:43 smithi002 ceph-mon[63804]: Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY) 2024-09-17T00:27:43.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:27:43 smithi002 ceph-mon[63804]: Health check failed: Degraded data redundancy: 19/726 objects degraded (2.617%), 4 pgs degraded (PG_DEGRADED) 2024-09-17T00:27:43.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:27:43 smithi002 ceph-mon[66732]: pgmap v154: 161 pgs: 7 active+undersized, 9 peering, 18 stale+active+clean, 4 active+undersized+degraded, 123 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 19/726 objects degraded (2.617%) 2024-09-17T00:27:43.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:27:43 smithi002 ceph-mon[66732]: Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY) 2024-09-17T00:27:43.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:27:43 smithi002 ceph-mon[66732]: Health check failed: Degraded data redundancy: 19/726 objects degraded (2.617%), 4 pgs degraded (PG_DEGRADED) 2024-09-17T00:27:43.967 INFO:journalctl@ceph.osd.4.smithi099.stdout:Sep 17 00:27:43 smithi099 podman[59714]: 2024-09-17 00:27:43.489889494 +0000 UTC m=+0.412270477 container init e73cc5df988f3d90794083afb65371aa84db58f481d90fec71ae809136875237 (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-4-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=squid-c2ddf97, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-09-17T00:27:43.968 INFO:journalctl@ceph.osd.4.smithi099.stdout:Sep 17 00:27:43 smithi099 podman[59714]: 2024-09-17 00:27:43.493351483 +0000 UTC m=+0.415732455 container start e73cc5df988f3d90794083afb65371aa84db58f481d90fec71ae809136875237 (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-4-activate, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=squid-c2ddf97, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-17T00:27:43.968 INFO:journalctl@ceph.osd.4.smithi099.stdout:Sep 17 00:27:43 smithi099 podman[59714]: 2024-09-17 00:27:43.573307555 +0000 UTC m=+0.495688539 container attach e73cc5df988f3d90794083afb65371aa84db58f481d90fec71ae809136875237 (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-4-activate, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=squid-c2ddf97, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True) 2024-09-17T00:27:43.968 INFO:journalctl@ceph.osd.4.smithi099.stdout:Sep 17 00:27:43 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-4-activate[59729]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-17T00:27:43.968 INFO:journalctl@ceph.osd.4.smithi099.stdout:Sep 17 00:27:43 smithi099 bash[59714]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-17T00:27:43.968 INFO:journalctl@ceph.osd.4.smithi099.stdout:Sep 17 00:27:43 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-4-activate[59729]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-17T00:27:43.968 INFO:journalctl@ceph.osd.4.smithi099.stdout:Sep 17 00:27:43 smithi099 bash[59714]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-17T00:27:45.273 INFO:journalctl@ceph.osd.4.smithi099.stdout:Sep 17 00:27:45 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-4-activate[59729]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-09-17T00:27:45.274 INFO:journalctl@ceph.osd.4.smithi099.stdout:Sep 17 00:27:45 smithi099 bash[59714]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-09-17T00:27:45.274 INFO:journalctl@ceph.osd.4.smithi099.stdout:Sep 17 00:27:45 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-4-activate[59729]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-09-17T00:27:45.274 INFO:journalctl@ceph.osd.4.smithi099.stdout:Sep 17 00:27:45 smithi099 bash[59714]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-09-17T00:27:45.632 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:27:45 smithi099 ceph-mon[56712]: pgmap v155: 161 pgs: 35 active+undersized, 9 peering, 3 stale+active+clean, 18 active+undersized+degraded, 96 active+clean; 585 KiB data, 179 MiB used, 715 GiB / 715 GiB avail; 88/726 objects degraded (12.121%) 2024-09-17T00:27:45.632 INFO:journalctl@ceph.osd.4.smithi099.stdout:Sep 17 00:27:45 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-4-activate[59729]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-09-17T00:27:45.633 INFO:journalctl@ceph.osd.4.smithi099.stdout:Sep 17 00:27:45 smithi099 bash[59714]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-09-17T00:27:45.633 INFO:journalctl@ceph.osd.4.smithi099.stdout:Sep 17 00:27:45 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-4-activate[59729]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-09-17T00:27:45.633 INFO:journalctl@ceph.osd.4.smithi099.stdout:Sep 17 00:27:45 smithi099 bash[59714]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-09-17T00:27:45.633 INFO:journalctl@ceph.osd.4.smithi099.stdout:Sep 17 00:27:45 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-4-activate[59729]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-4/block 2024-09-17T00:27:45.633 INFO:journalctl@ceph.osd.4.smithi099.stdout:Sep 17 00:27:45 smithi099 bash[59714]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-4/block 2024-09-17T00:27:45.633 INFO:journalctl@ceph.osd.4.smithi099.stdout:Sep 17 00:27:45 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-4-activate[59729]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-09-17T00:27:45.633 INFO:journalctl@ceph.osd.4.smithi099.stdout:Sep 17 00:27:45 smithi099 bash[59714]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-09-17T00:27:45.633 INFO:journalctl@ceph.osd.4.smithi099.stdout:Sep 17 00:27:45 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-4-activate[59729]: --> ceph-volume raw activate successful for osd ID: 4 2024-09-17T00:27:45.633 INFO:journalctl@ceph.osd.4.smithi099.stdout:Sep 17 00:27:45 smithi099 bash[59714]: --> ceph-volume raw activate successful for osd ID: 4 2024-09-17T00:27:45.633 INFO:journalctl@ceph.osd.4.smithi099.stdout:Sep 17 00:27:45 smithi099 podman[59714]: 2024-09-17 00:27:45.333030994 +0000 UTC m=+2.255411970 container died e73cc5df988f3d90794083afb65371aa84db58f481d90fec71ae809136875237 (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-4-activate, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, RELEASE=squid-c2ddf97, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-09-17T00:27:45.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:27:45 smithi002 ceph-mon[66732]: pgmap v155: 161 pgs: 35 active+undersized, 9 peering, 3 stale+active+clean, 18 active+undersized+degraded, 96 active+clean; 585 KiB data, 179 MiB used, 715 GiB / 715 GiB avail; 88/726 objects degraded (12.121%) 2024-09-17T00:27:45.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:27:45 smithi002 ceph-mon[63804]: pgmap v155: 161 pgs: 35 active+undersized, 9 peering, 3 stale+active+clean, 18 active+undersized+degraded, 96 active+clean; 585 KiB data, 179 MiB used, 715 GiB / 715 GiB avail; 88/726 objects degraded (12.121%) 2024-09-17T00:27:46.217 INFO:journalctl@ceph.osd.4.smithi099.stdout:Sep 17 00:27:45 smithi099 podman[59858]: 2024-09-17 00:27:45.92742921 +0000 UTC m=+0.583283412 container remove e73cc5df988f3d90794083afb65371aa84db58f481d90fec71ae809136875237 (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-4-activate, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, RELEASE=squid-c2ddf97, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-17T00:27:46.217 INFO:journalctl@ceph.osd.4.smithi099.stdout:Sep 17 00:27:46 smithi099 podman[59916]: 2024-09-17 00:27:46.050896731 +0000 UTC m=+0.016765757 image pull ebb0615149def4a3179c82c0cc47af75808300abb9f1142788e5e605a7ac0e5b quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f 2024-09-17T00:27:46.217 INFO:journalctl@ceph.osd.4.smithi099.stdout:Sep 17 00:27:46 smithi099 podman[59916]: 2024-09-17 00:27:46.153414264 +0000 UTC m=+0.119283281 container create f644e5be4db8ad107c096c308c1c5c59a9750602e759a4fdb7b95da1b60a89ec (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-4, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-c2ddf97) 2024-09-17T00:27:46.717 INFO:journalctl@ceph.osd.4.smithi099.stdout:Sep 17 00:27:46 smithi099 podman[59916]: 2024-09-17 00:27:46.389474104 +0000 UTC m=+0.355343127 container init f644e5be4db8ad107c096c308c1c5c59a9750602e759a4fdb7b95da1b60a89ec (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-4, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=squid-c2ddf97, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux ) 2024-09-17T00:27:46.717 INFO:journalctl@ceph.osd.4.smithi099.stdout:Sep 17 00:27:46 smithi099 podman[59916]: 2024-09-17 00:27:46.392848689 +0000 UTC m=+0.358717706 container start f644e5be4db8ad107c096c308c1c5c59a9750602e759a4fdb7b95da1b60a89ec (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-4, org.label-schema.build-date=20240731, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, RELEASE=squid-c2ddf97, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-09-17T00:27:46.717 INFO:journalctl@ceph.osd.4.smithi099.stdout:Sep 17 00:27:46 smithi099 ceph-osd[59935]: -- 172.21.15.99:0/826583473 <== mon.2 v2:172.21.15.99:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5633b1d03a40 con 0x5633b2aca400 2024-09-17T00:27:46.717 INFO:journalctl@ceph.osd.4.smithi099.stdout:Sep 17 00:27:46 smithi099 bash[59916]: f644e5be4db8ad107c096c308c1c5c59a9750602e759a4fdb7b95da1b60a89ec 2024-09-17T00:27:46.717 INFO:journalctl@ceph.osd.4.smithi099.stdout:Sep 17 00:27:46 smithi099 systemd[1]: Started Ceph osd.4 for 3aad27a2-7489-11ef-bceb-c7b262605968. 2024-09-17T00:27:46.979 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:27:46 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: ::ffff:172.21.15.99 - - [17/Sep/2024:00:27:46] "GET /metrics HTTP/1.1" 200 38445 "" "Prometheus/2.43.0" 2024-09-17T00:27:47.920 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:27:47 smithi099 ceph-mon[56712]: pgmap v156: 161 pgs: 38 active+undersized, 9 peering, 21 active+undersized+degraded, 93 active+clean; 585 KiB data, 179 MiB used, 715 GiB / 715 GiB avail; 97/726 objects degraded (13.361%) 2024-09-17T00:27:47.920 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:27:47 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:27:47.920 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:27:47 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:27:47.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:27:47 smithi002 ceph-mon[63804]: pgmap v156: 161 pgs: 38 active+undersized, 9 peering, 21 active+undersized+degraded, 93 active+clean; 585 KiB data, 179 MiB used, 715 GiB / 715 GiB avail; 97/726 objects degraded (13.361%) 2024-09-17T00:27:47.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:27:47 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:27:47.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:27:47 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:27:47.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:27:47 smithi002 ceph-mon[66732]: pgmap v156: 161 pgs: 38 active+undersized, 9 peering, 21 active+undersized+degraded, 93 active+clean; 585 KiB data, 179 MiB used, 715 GiB / 715 GiB avail; 97/726 objects degraded (13.361%) 2024-09-17T00:27:47.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:27:47 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:27:47.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:27:47 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:27:48.467 INFO:journalctl@ceph.osd.4.smithi099.stdout:Sep 17 00:27:48 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-4[59931]: 2024-09-17T00:27:48.277+0000 7f4626bad640 -1 Falling back to public interface 2024-09-17T00:27:48.864 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:27:48 smithi099 ceph-mon[56712]: Health check update: Degraded data redundancy: 97/726 objects degraded (13.361%), 21 pgs degraded (PG_DEGRADED) 2024-09-17T00:27:48.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:27:48 smithi002 ceph-mon[63804]: Health check update: Degraded data redundancy: 97/726 objects degraded (13.361%), 21 pgs degraded (PG_DEGRADED) 2024-09-17T00:27:48.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:27:48 smithi002 ceph-mon[66732]: Health check update: Degraded data redundancy: 97/726 objects degraded (13.361%), 21 pgs degraded (PG_DEGRADED) 2024-09-17T00:27:49.569 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:27:49 smithi099 ceph-mon[56712]: pgmap v157: 161 pgs: 42 active+undersized, 26 active+undersized+degraded, 93 active+clean; 585 KiB data, 179 MiB used, 715 GiB / 715 GiB avail; 124/726 objects degraded (17.080%) 2024-09-17T00:27:49.569 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:27:49 smithi099 ceph-mon[56712]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs peering) 2024-09-17T00:27:49.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:27:49 smithi002 ceph-mon[63804]: pgmap v157: 161 pgs: 42 active+undersized, 26 active+undersized+degraded, 93 active+clean; 585 KiB data, 179 MiB used, 715 GiB / 715 GiB avail; 124/726 objects degraded (17.080%) 2024-09-17T00:27:49.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:27:49 smithi002 ceph-mon[63804]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs peering) 2024-09-17T00:27:49.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:27:49 smithi002 ceph-mon[66732]: pgmap v157: 161 pgs: 42 active+undersized, 26 active+undersized+degraded, 93 active+clean; 585 KiB data, 179 MiB used, 715 GiB / 715 GiB avail; 124/726 objects degraded (17.080%) 2024-09-17T00:27:49.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:27:49 smithi002 ceph-mon[66732]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs peering) 2024-09-17T00:27:51.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:27:51 smithi002 ceph-mon[66732]: pgmap v158: 161 pgs: 42 active+undersized, 26 active+undersized+degraded, 93 active+clean; 585 KiB data, 179 MiB used, 715 GiB / 715 GiB avail; 124/726 objects degraded (17.080%) 2024-09-17T00:27:51.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:27:51 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:27:51.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:27:51 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:27:51.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:27:51 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:27:51.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:27:51 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:27:51.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:27:51 smithi002 ceph-mon[63804]: pgmap v158: 161 pgs: 42 active+undersized, 26 active+undersized+degraded, 93 active+clean; 585 KiB data, 179 MiB used, 715 GiB / 715 GiB avail; 124/726 objects degraded (17.080%) 2024-09-17T00:27:51.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:27:51 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:27:51.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:27:51 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:27:51.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:27:51 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:27:51.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:27:51 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:27:51.809 INFO:journalctl@ceph.osd.4.smithi099.stdout:Sep 17 00:27:51 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-4[59931]: 2024-09-17T00:27:51.783+0000 7f4626bad640 -1 osd.4 0 read_superblock omap replica is missing. 2024-09-17T00:27:51.810 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:27:51 smithi099 ceph-mon[56712]: pgmap v158: 161 pgs: 42 active+undersized, 26 active+undersized+degraded, 93 active+clean; 585 KiB data, 179 MiB used, 715 GiB / 715 GiB avail; 124/726 objects degraded (17.080%) 2024-09-17T00:27:51.810 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:27:51 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:27:51.810 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:27:51 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:27:51.810 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:27:51 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:27:51.810 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:27:51 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:27:52.080 INFO:journalctl@ceph.osd.4.smithi099.stdout:Sep 17 00:27:51 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-4[59931]: 2024-09-17T00:27:51.812+0000 7f4626bad640 -1 osd.4 132 log_to_monitors true 2024-09-17T00:27:52.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:27:52 smithi099 ceph-mon[56712]: from='client.44563 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:27:52.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:27:52 smithi099 ceph-mon[56712]: from='osd.4 [v2:172.21.15.99:6800/2022983386,v1:172.21.15.99:6801/2022983386]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-17T00:27:52.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:27:52 smithi099 ceph-mon[56712]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-17T00:27:52.967 INFO:journalctl@ceph.osd.4.smithi099.stdout:Sep 17 00:27:52 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-4[59931]: 2024-09-17T00:27:52.578+0000 7f461e958640 -1 osd.4 132 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T00:27:52.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:27:52 smithi002 ceph-mon[63804]: from='client.44563 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:27:52.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:27:52 smithi002 ceph-mon[63804]: from='osd.4 [v2:172.21.15.99:6800/2022983386,v1:172.21.15.99:6801/2022983386]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-17T00:27:52.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:27:52 smithi002 ceph-mon[63804]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-17T00:27:52.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:27:52 smithi002 ceph-mon[66732]: from='client.44563 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:27:52.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:27:52 smithi002 ceph-mon[66732]: from='osd.4 [v2:172.21.15.99:6800/2022983386,v1:172.21.15.99:6801/2022983386]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-17T00:27:52.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:27:52 smithi002 ceph-mon[66732]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-17T00:27:53.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:27:53 smithi099 ceph-mon[56712]: pgmap v159: 161 pgs: 42 active+undersized, 26 active+undersized+degraded, 93 active+clean; 585 KiB data, 179 MiB used, 715 GiB / 715 GiB avail; 124/726 objects degraded (17.080%) 2024-09-17T00:27:53.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:27:53 smithi099 ceph-mon[56712]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-17T00:27:53.968 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:27:53 smithi099 ceph-mon[56712]: from='osd.4 [v2:172.21.15.99:6800/2022983386,v1:172.21.15.99:6801/2022983386]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi099", "root=default"]}]: dispatch 2024-09-17T00:27:53.968 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:27:53 smithi099 ceph-mon[56712]: osdmap e135: 8 total, 7 up, 8 in 2024-09-17T00:27:53.968 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:27:53 smithi099 ceph-mon[56712]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi099", "root=default"]}]: dispatch 2024-09-17T00:27:53.968 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:27:53 smithi099 ceph-mon[56712]: Health check update: Degraded data redundancy: 124/726 objects degraded (17.080%), 26 pgs degraded (PG_DEGRADED) 2024-09-17T00:27:53.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:27:53 smithi002 ceph-mon[63804]: pgmap v159: 161 pgs: 42 active+undersized, 26 active+undersized+degraded, 93 active+clean; 585 KiB data, 179 MiB used, 715 GiB / 715 GiB avail; 124/726 objects degraded (17.080%) 2024-09-17T00:27:53.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:27:53 smithi002 ceph-mon[63804]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-17T00:27:53.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:27:53 smithi002 ceph-mon[63804]: from='osd.4 [v2:172.21.15.99:6800/2022983386,v1:172.21.15.99:6801/2022983386]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi099", "root=default"]}]: dispatch 2024-09-17T00:27:53.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:27:53 smithi002 ceph-mon[63804]: osdmap e135: 8 total, 7 up, 8 in 2024-09-17T00:27:53.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:27:53 smithi002 ceph-mon[63804]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi099", "root=default"]}]: dispatch 2024-09-17T00:27:53.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:27:53 smithi002 ceph-mon[63804]: Health check update: Degraded data redundancy: 124/726 objects degraded (17.080%), 26 pgs degraded (PG_DEGRADED) 2024-09-17T00:27:53.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:27:53 smithi002 ceph-mon[66732]: pgmap v159: 161 pgs: 42 active+undersized, 26 active+undersized+degraded, 93 active+clean; 585 KiB data, 179 MiB used, 715 GiB / 715 GiB avail; 124/726 objects degraded (17.080%) 2024-09-17T00:27:53.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:27:53 smithi002 ceph-mon[66732]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-17T00:27:53.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:27:53 smithi002 ceph-mon[66732]: from='osd.4 [v2:172.21.15.99:6800/2022983386,v1:172.21.15.99:6801/2022983386]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi099", "root=default"]}]: dispatch 2024-09-17T00:27:53.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:27:53 smithi002 ceph-mon[66732]: osdmap e135: 8 total, 7 up, 8 in 2024-09-17T00:27:53.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:27:53 smithi002 ceph-mon[66732]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi099", "root=default"]}]: dispatch 2024-09-17T00:27:53.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:27:53 smithi002 ceph-mon[66732]: Health check update: Degraded data redundancy: 124/726 objects degraded (17.080%), 26 pgs degraded (PG_DEGRADED) 2024-09-17T00:27:54.694 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:27:54 smithi099 ceph-mon[56712]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-17T00:27:54.694 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:27:54 smithi099 ceph-mon[56712]: osd.4 [v2:172.21.15.99:6800/2022983386,v1:172.21.15.99:6801/2022983386] boot 2024-09-17T00:27:54.694 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:27:54 smithi099 ceph-mon[56712]: osdmap e136: 8 total, 8 up, 8 in 2024-09-17T00:27:54.694 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:27:54 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T00:27:54.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:27:54 smithi002 ceph-mon[66732]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-17T00:27:54.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:27:54 smithi002 ceph-mon[66732]: osd.4 [v2:172.21.15.99:6800/2022983386,v1:172.21.15.99:6801/2022983386] boot 2024-09-17T00:27:54.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:27:54 smithi002 ceph-mon[66732]: osdmap e136: 8 total, 8 up, 8 in 2024-09-17T00:27:54.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:27:54 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T00:27:54.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:27:54 smithi002 ceph-mon[63804]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-17T00:27:54.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:27:54 smithi002 ceph-mon[63804]: osd.4 [v2:172.21.15.99:6800/2022983386,v1:172.21.15.99:6801/2022983386] boot 2024-09-17T00:27:54.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:27:54 smithi002 ceph-mon[63804]: osdmap e136: 8 total, 8 up, 8 in 2024-09-17T00:27:54.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:27:54 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-17T00:27:55.868 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:27:55 smithi099 ceph-mon[56712]: pgmap v162: 161 pgs: 4 peering, 39 active+undersized, 25 active+undersized+degraded, 93 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 119/726 objects degraded (16.391%) 2024-09-17T00:27:55.869 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:27:55 smithi099 ceph-mon[56712]: osdmap e137: 8 total, 8 up, 8 in 2024-09-17T00:27:55.869 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:27:55 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:27:55.869 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:27:55 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:27:55.869 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:27:55 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:27:55.869 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:27:55 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:27:55.869 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:27:55 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:27:55.869 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:27:55 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi002.vmouem", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-17T00:27:55.869 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:27:55 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi002.vmouem", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-17T00:27:55.869 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:27:55 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:27:55.968 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:27:55 smithi002 ceph-mon[66732]: pgmap v162: 161 pgs: 4 peering, 39 active+undersized, 25 active+undersized+degraded, 93 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 119/726 objects degraded (16.391%) 2024-09-17T00:27:55.968 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:27:55 smithi002 ceph-mon[66732]: osdmap e137: 8 total, 8 up, 8 in 2024-09-17T00:27:55.968 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:27:55 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:27:55.969 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:27:55 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:27:55.969 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:27:55 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:27:55.969 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:27:55 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:27:55.969 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:27:55 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:27:55.969 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:27:55 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi002.vmouem", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-17T00:27:55.969 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:27:55 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi002.vmouem", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-17T00:27:55.969 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:27:55 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:27:55.969 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:27:55 smithi002 ceph-mon[63804]: pgmap v162: 161 pgs: 4 peering, 39 active+undersized, 25 active+undersized+degraded, 93 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 119/726 objects degraded (16.391%) 2024-09-17T00:27:55.969 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:27:55 smithi002 ceph-mon[63804]: osdmap e137: 8 total, 8 up, 8 in 2024-09-17T00:27:55.969 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:27:55 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:27:55.969 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:27:55 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:27:55.970 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:27:55 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:27:55.970 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:27:55 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:27:55.970 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:27:55 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:27:55.970 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:27:55 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi002.vmouem", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-17T00:27:55.970 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:27:55 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi002.vmouem", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-17T00:27:55.970 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:27:55 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:27:56.641 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:27:56 smithi002 ceph-mon[66732]: Reconfiguring iscsi.foo.smithi002.vmouem (dependencies changed)... 2024-09-17T00:27:56.641 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:27:56 smithi002 ceph-mon[66732]: Reconfiguring daemon iscsi.foo.smithi002.vmouem on smithi002 2024-09-17T00:27:56.641 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:27:56 smithi002 ceph-mon[63804]: Reconfiguring iscsi.foo.smithi002.vmouem (dependencies changed)... 2024-09-17T00:27:56.641 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:27:56 smithi002 ceph-mon[63804]: Reconfiguring daemon iscsi.foo.smithi002.vmouem on smithi002 2024-09-17T00:27:56.885 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:27:56 smithi099 ceph-mon[56712]: Reconfiguring iscsi.foo.smithi002.vmouem (dependencies changed)... 2024-09-17T00:27:56.886 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:27:56 smithi099 ceph-mon[56712]: Reconfiguring daemon iscsi.foo.smithi002.vmouem on smithi002 2024-09-17T00:27:56.963 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:27:56 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: ::ffff:172.21.15.99 - - [17/Sep/2024:00:27:56] "GET /metrics HTTP/1.1" 200 38445 "" "Prometheus/2.43.0" 2024-09-17T00:27:57.838 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:27:57 smithi002 ceph-mon[63804]: pgmap v164: 161 pgs: 3 activating, 4 peering, 37 active+undersized, 24 active+undersized+degraded, 93 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 118/726 objects degraded (16.253%) 2024-09-17T00:27:57.839 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:27:57 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:27:57.839 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:27:57 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T00:27:57.839 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:27:57 smithi002 ceph-mon[66732]: pgmap v164: 161 pgs: 3 activating, 4 peering, 37 active+undersized, 24 active+undersized+degraded, 93 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 118/726 objects degraded (16.253%) 2024-09-17T00:27:57.839 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:27:57 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:27:57.840 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:27:57 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T00:27:57.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:27:57 smithi099 ceph-mon[56712]: pgmap v164: 161 pgs: 3 activating, 4 peering, 37 active+undersized, 24 active+undersized+degraded, 93 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 118/726 objects degraded (16.253%) 2024-09-17T00:27:57.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:27:57 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:27:57.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:27:57 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T00:27:58.230 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:27:57 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: 2024-09-17T00:27:57.840+0000 7f17e1e2b640 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi002.vmouem ... 2024-09-17T00:27:58.230 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:27:57 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: Creating ceph-iscsi config... 2024-09-17T00:27:58.230 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:27:57 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/iscsi-gateway.cfg 2024-09-17T00:27:58.230 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:27:57 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/tcmu-runner-entrypoint.sh 2024-09-17T00:27:58.230 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:27:57 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: Non-zero exit code 1 from systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem 2024-09-17T00:27:58.230 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:27:57 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: systemctl: stderr Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:27:58.230 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:27:57 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: systemctl: stderr See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:27:58.231 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:27:57 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: Traceback (most recent call last): 2024-09-17T00:27:58.231 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:27:57 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-17T00:27:58.231 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:27:57 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: return _run_code(code, main_globals, None, 2024-09-17T00:27:58.231 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:27:57 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-17T00:27:58.231 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:27:57 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: exec(code, run_globals) 2024-09-17T00:27:58.231 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:27:57 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5579, in 2024-09-17T00:27:58.231 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:27:57 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5567, in main 2024-09-17T00:27:58.231 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:27:57 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3049, in command_deploy_from 2024-09-17T00:27:58.232 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:27:57 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3084, in _common_deploy 2024-09-17T00:27:58.232 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:27:57 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3104, in _deploy_daemon_container 2024-09-17T00:27:58.232 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:27:57 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 1124, in deploy_daemon 2024-09-17T00:27:58.232 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:27:57 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/tmp/tmpnroy8d0w.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-17T00:27:58.232 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:27:57 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: RuntimeError: Failed command: systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem: Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:27:58.232 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:27:57 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:27:58.233 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:27:57 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: Traceback (most recent call last): 2024-09-17T00:27:58.233 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:27:57 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1134, in _check_daemons 2024-09-17T00:27:58.233 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:27:57 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: self.mgr._daemon_action(daemon_spec, action=action) 2024-09-17T00:27:58.233 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:27:57 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2473, in _daemon_action 2024-09-17T00:27:58.233 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:27:57 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: return self.wait_async( 2024-09-17T00:27:58.234 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:27:57 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/usr/share/ceph/mgr/cephadm/module.py", line 796, in wait_async 2024-09-17T00:27:58.234 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:27:57 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: return self.event_loop.get_result(coro, timeout) 2024-09-17T00:27:58.234 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:27:57 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-09-17T00:27:58.234 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:27:57 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: return future.result(timeout) 2024-09-17T00:27:58.234 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:27:57 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/lib64/python3.9/concurrent/futures/_base.py", line 446, in result 2024-09-17T00:27:58.234 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:27:57 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: return self.__get_result() 2024-09-17T00:27:58.234 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:27:57 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/lib64/python3.9/concurrent/futures/_base.py", line 391, in __get_result 2024-09-17T00:27:58.234 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:27:57 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: raise self._exception 2024-09-17T00:27:58.234 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:27:57 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1370, in _create_daemon 2024-09-17T00:27:58.234 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:27:57 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: out, err, code = await self._run_cephadm( 2024-09-17T00:27:58.234 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:27:57 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1713, in _run_cephadm 2024-09-17T00:27:58.234 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:27:57 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: raise OrchestratorError( 2024-09-17T00:27:58.234 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:27:57 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi002.vmouem ... 2024-09-17T00:27:58.234 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:27:57 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: Creating ceph-iscsi config... 2024-09-17T00:27:58.235 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:27:57 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/iscsi-gateway.cfg 2024-09-17T00:27:58.235 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:27:57 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/tcmu-runner-entrypoint.sh 2024-09-17T00:27:58.235 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:27:57 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: Non-zero exit code 1 from systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem 2024-09-17T00:27:58.235 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:27:57 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: systemctl: stderr Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:27:58.235 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:27:57 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: systemctl: stderr See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:27:58.235 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:27:57 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: Traceback (most recent call last): 2024-09-17T00:27:58.235 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:27:57 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-17T00:27:58.235 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:27:57 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: return _run_code(code, main_globals, None, 2024-09-17T00:27:58.235 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:27:57 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-17T00:27:58.235 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:27:57 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: exec(code, run_globals) 2024-09-17T00:27:58.235 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:27:57 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5579, in 2024-09-17T00:27:58.235 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:27:57 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5567, in main 2024-09-17T00:27:58.235 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:27:57 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3049, in command_deploy_from 2024-09-17T00:27:58.236 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:27:57 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3084, in _common_deploy 2024-09-17T00:27:58.236 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:27:57 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3104, in _deploy_daemon_container 2024-09-17T00:27:58.236 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:27:57 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 1124, in deploy_daemon 2024-09-17T00:27:58.236 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:27:57 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: File "/tmp/tmpnroy8d0w.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-17T00:27:58.236 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:27:57 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: RuntimeError: Failed command: systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem: Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:27:58.236 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:27:57 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:27:58.236 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:27:57 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: 2024-09-17T00:27:57.888+0000 7f17efec7640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (13 PGs are or would become offline) 2024-09-17T00:27:58.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:27:58 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T00:27:58.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:27:58 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:27:58.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:27:58 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:27:58.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:27:58 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:27:58.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:27:58 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-09-17T00:27:58.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:27:58 smithi099 ceph-mon[56712]: Health check update: Degraded data redundancy: 118/726 objects degraded (16.253%), 24 pgs degraded (PG_DEGRADED) 2024-09-17T00:27:58.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:27:58 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T00:27:58.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:27:58 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:27:58.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:27:58 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:27:58.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:27:58 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:27:58.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:27:58 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-09-17T00:27:58.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:27:58 smithi002 ceph-mon[66732]: Health check update: Degraded data redundancy: 118/726 objects degraded (16.253%), 24 pgs degraded (PG_DEGRADED) 2024-09-17T00:27:58.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:27:58 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T00:27:58.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:27:58 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:27:58.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:27:58 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:27:58.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:27:58 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:27:58.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:27:58 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-09-17T00:27:58.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:27:58 smithi002 ceph-mon[63804]: Health check update: Degraded data redundancy: 118/726 objects degraded (16.253%), 24 pgs degraded (PG_DEGRADED) 2024-09-17T00:27:59.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:27:59 smithi099 ceph-mon[56712]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi002.vmouem ... 2024-09-17T00:27:59.967 INFO:journalctl@ceph.mon.b.smithi099.stdout: Creating ceph-iscsi config... 2024-09-17T00:27:59.967 INFO:journalctl@ceph.mon.b.smithi099.stdout: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/iscsi-gateway.cfg 2024-09-17T00:27:59.967 INFO:journalctl@ceph.mon.b.smithi099.stdout: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/tcmu-runner-entrypoint.sh 2024-09-17T00:27:59.968 INFO:journalctl@ceph.mon.b.smithi099.stdout: Non-zero exit code 1 from systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem 2024-09-17T00:27:59.968 INFO:journalctl@ceph.mon.b.smithi099.stdout: systemctl: stderr Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:27:59.968 INFO:journalctl@ceph.mon.b.smithi099.stdout: systemctl: stderr See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:27:59.968 INFO:journalctl@ceph.mon.b.smithi099.stdout: Traceback (most recent call last): 2024-09-17T00:27:59.968 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-17T00:27:59.968 INFO:journalctl@ceph.mon.b.smithi099.stdout: return _run_code(code, main_globals, None, 2024-09-17T00:27:59.968 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-17T00:27:59.968 INFO:journalctl@ceph.mon.b.smithi099.stdout: exec(code, run_globals) 2024-09-17T00:27:59.968 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5579, in 2024-09-17T00:27:59.968 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5567, in main 2024-09-17T00:27:59.968 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3049, in command_deploy_from 2024-09-17T00:27:59.968 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3084, in _common_deploy 2024-09-17T00:27:59.969 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3104, in _deploy_daemon_container 2024-09-17T00:27:59.969 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 1124, in deploy_daemon 2024-09-17T00:27:59.969 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-17T00:27:59.969 INFO:journalctl@ceph.mon.b.smithi099.stdout: RuntimeError: Failed command: systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem: Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:27:59.969 INFO:journalctl@ceph.mon.b.smithi099.stdout: See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:27:59.969 INFO:journalctl@ceph.mon.b.smithi099.stdout: Traceback (most recent call last): 2024-09-17T00:27:59.969 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1134, in _check_daemons 2024-09-17T00:27:59.969 INFO:journalctl@ceph.mon.b.smithi099.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-09-17T00:27:59.969 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2473, in _daemon_action 2024-09-17T00:27:59.969 INFO:journalctl@ceph.mon.b.smithi099.stdout: return self.wait_async( 2024-09-17T00:27:59.969 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 796, in wait_async 2024-09-17T00:27:59.969 INFO:journalctl@ceph.mon.b.smithi099.stdout: return self.event_loop.get_result(coro, timeout) 2024-09-17T00:27:59.969 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-09-17T00:27:59.969 INFO:journalctl@ceph.mon.b.smithi099.stdout: return future.result(timeout) 2024-09-17T00:27:59.970 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/lib64/python3.9/concurrent/futures/_base.py", line 446, in result 2024-09-17T00:27:59.970 INFO:journalctl@ceph.mon.b.smithi099.stdout: return self.__get_result() 2024-09-17T00:27:59.970 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/lib64/python3.9/concurrent/futures/_base.py", line 391, in __get_result 2024-09-17T00:27:59.970 INFO:journalctl@ceph.mon.b.smithi099.stdout: raise self._exception 2024-09-17T00:27:59.970 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1370, in _create_daemon 2024-09-17T00:27:59.970 INFO:journalctl@ceph.mon.b.smithi099.stdout: out, err, code = await self._run_cephadm( 2024-09-17T00:27:59.970 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1713, in _run_cephadm 2024-09-17T00:27:59.970 INFO:journalctl@ceph.mon.b.smithi099.stdout: raise OrchestratorError( 2024-09-17T00:27:59.970 INFO:journalctl@ceph.mon.b.smithi099.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi002.vmouem ... 2024-09-17T00:27:59.970 INFO:journalctl@ceph.mon.b.smithi099.stdout: Creating ceph-iscsi config... 2024-09-17T00:27:59.970 INFO:journalctl@ceph.mon.b.smithi099.stdout: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/iscsi-gateway.cfg 2024-09-17T00:27:59.970 INFO:journalctl@ceph.mon.b.smithi099.stdout: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/tcmu-runner-entrypoint.sh 2024-09-17T00:27:59.970 INFO:journalctl@ceph.mon.b.smithi099.stdout: Non-zero exit code 1 from systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem 2024-09-17T00:27:59.970 INFO:journalctl@ceph.mon.b.smithi099.stdout: systemctl: stderr Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:27:59.970 INFO:journalctl@ceph.mon.b.smithi099.stdout: systemctl: stderr See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:27:59.971 INFO:journalctl@ceph.mon.b.smithi099.stdout: Traceback (most recent call last): 2024-09-17T00:27:59.971 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-17T00:27:59.971 INFO:journalctl@ceph.mon.b.smithi099.stdout: return _run_code(code, main_globals, None, 2024-09-17T00:27:59.971 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-17T00:27:59.971 INFO:journalctl@ceph.mon.b.smithi099.stdout: exec(code, run_globals) 2024-09-17T00:27:59.971 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5579, in 2024-09-17T00:27:59.971 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5567, in main 2024-09-17T00:27:59.971 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3049, in command_deploy_from 2024-09-17T00:27:59.971 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3084, in _common_deploy 2024-09-17T00:27:59.971 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3104, in _deploy_daemon_container 2024-09-17T00:27:59.971 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 1124, in deploy_daemon 2024-09-17T00:27:59.971 INFO:journalctl@ceph.mon.b.smithi099.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-17T00:27:59.971 INFO:journalctl@ceph.mon.b.smithi099.stdout: RuntimeError: Failed command: systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem: Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:27:59.971 INFO:journalctl@ceph.mon.b.smithi099.stdout: See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:27:59.972 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:27:59 smithi099 ceph-mon[56712]: from='mon.1 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-09-17T00:27:59.972 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:27:59 smithi099 ceph-mon[56712]: Upgrade: unsafe to stop osd(s) at this time (13 PGs are or would become offline) 2024-09-17T00:27:59.972 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:27:59 smithi099 ceph-mon[56712]: pgmap v165: 161 pgs: 3 activating, 4 peering, 154 active+clean; 585 KiB data, 194 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:27:59.972 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:27:59 smithi099 ceph-mon[56712]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 118/726 objects degraded (16.253%), 24 pgs degraded) 2024-09-17T00:27:59.972 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:27:59 smithi099 ceph-mon[56712]: Cluster is now healthy 2024-09-17T00:27:59.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:27:59 smithi002 ceph-mon[63804]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi002.vmouem ... 2024-09-17T00:27:59.979 INFO:journalctl@ceph.mon.a.smithi002.stdout: Creating ceph-iscsi config... 2024-09-17T00:27:59.979 INFO:journalctl@ceph.mon.a.smithi002.stdout: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/iscsi-gateway.cfg 2024-09-17T00:27:59.979 INFO:journalctl@ceph.mon.a.smithi002.stdout: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/tcmu-runner-entrypoint.sh 2024-09-17T00:27:59.979 INFO:journalctl@ceph.mon.a.smithi002.stdout: Non-zero exit code 1 from systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem 2024-09-17T00:27:59.979 INFO:journalctl@ceph.mon.a.smithi002.stdout: systemctl: stderr Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:27:59.979 INFO:journalctl@ceph.mon.a.smithi002.stdout: systemctl: stderr See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:27:59.980 INFO:journalctl@ceph.mon.a.smithi002.stdout: Traceback (most recent call last): 2024-09-17T00:27:59.980 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-17T00:27:59.980 INFO:journalctl@ceph.mon.a.smithi002.stdout: return _run_code(code, main_globals, None, 2024-09-17T00:27:59.980 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-17T00:27:59.980 INFO:journalctl@ceph.mon.a.smithi002.stdout: exec(code, run_globals) 2024-09-17T00:27:59.980 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5579, in 2024-09-17T00:27:59.980 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5567, in main 2024-09-17T00:27:59.980 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3049, in command_deploy_from 2024-09-17T00:27:59.980 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3084, in _common_deploy 2024-09-17T00:27:59.980 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3104, in _deploy_daemon_container 2024-09-17T00:27:59.980 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 1124, in deploy_daemon 2024-09-17T00:27:59.980 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-17T00:27:59.980 INFO:journalctl@ceph.mon.a.smithi002.stdout: RuntimeError: Failed command: systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem: Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:27:59.980 INFO:journalctl@ceph.mon.a.smithi002.stdout: See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:27:59.981 INFO:journalctl@ceph.mon.a.smithi002.stdout: Traceback (most recent call last): 2024-09-17T00:27:59.981 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1134, in _check_daemons 2024-09-17T00:27:59.981 INFO:journalctl@ceph.mon.a.smithi002.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-09-17T00:27:59.981 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2473, in _daemon_action 2024-09-17T00:27:59.981 INFO:journalctl@ceph.mon.a.smithi002.stdout: return self.wait_async( 2024-09-17T00:27:59.981 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 796, in wait_async 2024-09-17T00:27:59.981 INFO:journalctl@ceph.mon.a.smithi002.stdout: return self.event_loop.get_result(coro, timeout) 2024-09-17T00:27:59.981 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-09-17T00:27:59.981 INFO:journalctl@ceph.mon.a.smithi002.stdout: return future.result(timeout) 2024-09-17T00:27:59.981 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/lib64/python3.9/concurrent/futures/_base.py", line 446, in result 2024-09-17T00:27:59.981 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:27:59 smithi002 ceph-mon[66732]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi002.vmouem ... 2024-09-17T00:27:59.982 INFO:journalctl@ceph.mon.c.smithi002.stdout: Creating ceph-iscsi config... 2024-09-17T00:27:59.982 INFO:journalctl@ceph.mon.c.smithi002.stdout: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/iscsi-gateway.cfg 2024-09-17T00:27:59.982 INFO:journalctl@ceph.mon.c.smithi002.stdout: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/tcmu-runner-entrypoint.sh 2024-09-17T00:27:59.982 INFO:journalctl@ceph.mon.c.smithi002.stdout: Non-zero exit code 1 from systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem 2024-09-17T00:27:59.982 INFO:journalctl@ceph.mon.c.smithi002.stdout: systemctl: stderr Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:27:59.982 INFO:journalctl@ceph.mon.c.smithi002.stdout: systemctl: stderr See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:27:59.982 INFO:journalctl@ceph.mon.c.smithi002.stdout: Traceback (most recent call last): 2024-09-17T00:27:59.982 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-17T00:27:59.982 INFO:journalctl@ceph.mon.c.smithi002.stdout: return _run_code(code, main_globals, None, 2024-09-17T00:27:59.982 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-17T00:27:59.982 INFO:journalctl@ceph.mon.c.smithi002.stdout: exec(code, run_globals) 2024-09-17T00:27:59.982 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5579, in 2024-09-17T00:27:59.982 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5567, in main 2024-09-17T00:27:59.983 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3049, in command_deploy_from 2024-09-17T00:27:59.983 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3084, in _common_deploy 2024-09-17T00:27:59.983 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3104, in _deploy_daemon_container 2024-09-17T00:27:59.983 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 1124, in deploy_daemon 2024-09-17T00:27:59.983 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-17T00:27:59.983 INFO:journalctl@ceph.mon.c.smithi002.stdout: RuntimeError: Failed command: systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem: Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:27:59.983 INFO:journalctl@ceph.mon.c.smithi002.stdout: See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:27:59.983 INFO:journalctl@ceph.mon.c.smithi002.stdout: Traceback (most recent call last): 2024-09-17T00:27:59.983 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1134, in _check_daemons 2024-09-17T00:27:59.983 INFO:journalctl@ceph.mon.c.smithi002.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-09-17T00:27:59.983 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2473, in _daemon_action 2024-09-17T00:27:59.983 INFO:journalctl@ceph.mon.c.smithi002.stdout: return self.wait_async( 2024-09-17T00:27:59.983 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 796, in wait_async 2024-09-17T00:27:59.983 INFO:journalctl@ceph.mon.c.smithi002.stdout: return self.event_loop.get_result(coro, timeout) 2024-09-17T00:27:59.983 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-09-17T00:27:59.984 INFO:journalctl@ceph.mon.c.smithi002.stdout: return future.result(timeout) 2024-09-17T00:27:59.984 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/lib64/python3.9/concurrent/futures/_base.py", line 446, in result 2024-09-17T00:27:59.984 INFO:journalctl@ceph.mon.a.smithi002.stdout: return self.__get_result() 2024-09-17T00:27:59.984 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/lib64/python3.9/concurrent/futures/_base.py", line 391, in __get_result 2024-09-17T00:27:59.984 INFO:journalctl@ceph.mon.a.smithi002.stdout: raise self._exception 2024-09-17T00:27:59.984 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1370, in _create_daemon 2024-09-17T00:27:59.984 INFO:journalctl@ceph.mon.a.smithi002.stdout: out, err, code = await self._run_cephadm( 2024-09-17T00:27:59.984 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1713, in _run_cephadm 2024-09-17T00:27:59.984 INFO:journalctl@ceph.mon.a.smithi002.stdout: raise OrchestratorError( 2024-09-17T00:27:59.984 INFO:journalctl@ceph.mon.a.smithi002.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi002.vmouem ... 2024-09-17T00:27:59.985 INFO:journalctl@ceph.mon.a.smithi002.stdout: Creating ceph-iscsi config... 2024-09-17T00:27:59.985 INFO:journalctl@ceph.mon.a.smithi002.stdout: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/iscsi-gateway.cfg 2024-09-17T00:27:59.985 INFO:journalctl@ceph.mon.a.smithi002.stdout: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/tcmu-runner-entrypoint.sh 2024-09-17T00:27:59.985 INFO:journalctl@ceph.mon.a.smithi002.stdout: Non-zero exit code 1 from systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem 2024-09-17T00:27:59.985 INFO:journalctl@ceph.mon.a.smithi002.stdout: systemctl: stderr Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:27:59.985 INFO:journalctl@ceph.mon.a.smithi002.stdout: systemctl: stderr See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:27:59.985 INFO:journalctl@ceph.mon.a.smithi002.stdout: Traceback (most recent call last): 2024-09-17T00:27:59.985 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-17T00:27:59.985 INFO:journalctl@ceph.mon.a.smithi002.stdout: return _run_code(code, main_globals, None, 2024-09-17T00:27:59.985 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-17T00:27:59.985 INFO:journalctl@ceph.mon.a.smithi002.stdout: exec(code, run_globals) 2024-09-17T00:27:59.986 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5579, in 2024-09-17T00:27:59.986 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5567, in main 2024-09-17T00:27:59.986 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3049, in command_deploy_from 2024-09-17T00:27:59.986 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3084, in _common_deploy 2024-09-17T00:27:59.986 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3104, in _deploy_daemon_container 2024-09-17T00:27:59.987 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 1124, in deploy_daemon 2024-09-17T00:27:59.987 INFO:journalctl@ceph.mon.a.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-17T00:27:59.987 INFO:journalctl@ceph.mon.a.smithi002.stdout: RuntimeError: Failed command: systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem: Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:27:59.987 INFO:journalctl@ceph.mon.a.smithi002.stdout: See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:27:59.987 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:27:59 smithi002 ceph-mon[63804]: from='mon.1 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-09-17T00:27:59.987 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:27:59 smithi002 ceph-mon[63804]: Upgrade: unsafe to stop osd(s) at this time (13 PGs are or would become offline) 2024-09-17T00:27:59.988 INFO:journalctl@ceph.mon.c.smithi002.stdout: return self.__get_result() 2024-09-17T00:27:59.988 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/lib64/python3.9/concurrent/futures/_base.py", line 391, in __get_result 2024-09-17T00:27:59.988 INFO:journalctl@ceph.mon.c.smithi002.stdout: raise self._exception 2024-09-17T00:27:59.988 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1370, in _create_daemon 2024-09-17T00:27:59.988 INFO:journalctl@ceph.mon.c.smithi002.stdout: out, err, code = await self._run_cephadm( 2024-09-17T00:27:59.988 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1713, in _run_cephadm 2024-09-17T00:27:59.988 INFO:journalctl@ceph.mon.c.smithi002.stdout: raise OrchestratorError( 2024-09-17T00:27:59.988 INFO:journalctl@ceph.mon.c.smithi002.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi002.vmouem ... 2024-09-17T00:27:59.988 INFO:journalctl@ceph.mon.c.smithi002.stdout: Creating ceph-iscsi config... 2024-09-17T00:27:59.988 INFO:journalctl@ceph.mon.c.smithi002.stdout: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/iscsi-gateway.cfg 2024-09-17T00:27:59.988 INFO:journalctl@ceph.mon.c.smithi002.stdout: Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/tcmu-runner-entrypoint.sh 2024-09-17T00:27:59.988 INFO:journalctl@ceph.mon.c.smithi002.stdout: Non-zero exit code 1 from systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem 2024-09-17T00:27:59.988 INFO:journalctl@ceph.mon.c.smithi002.stdout: systemctl: stderr Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:27:59.989 INFO:journalctl@ceph.mon.c.smithi002.stdout: systemctl: stderr See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:27:59.989 INFO:journalctl@ceph.mon.c.smithi002.stdout: Traceback (most recent call last): 2024-09-17T00:27:59.989 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-17T00:27:59.989 INFO:journalctl@ceph.mon.c.smithi002.stdout: return _run_code(code, main_globals, None, 2024-09-17T00:27:59.989 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-17T00:27:59.989 INFO:journalctl@ceph.mon.c.smithi002.stdout: exec(code, run_globals) 2024-09-17T00:27:59.989 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5579, in 2024-09-17T00:27:59.989 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5567, in main 2024-09-17T00:27:59.989 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3049, in command_deploy_from 2024-09-17T00:27:59.989 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3084, in _common_deploy 2024-09-17T00:27:59.989 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3104, in _deploy_daemon_container 2024-09-17T00:27:59.989 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 1124, in deploy_daemon 2024-09-17T00:27:59.989 INFO:journalctl@ceph.mon.c.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-17T00:27:59.989 INFO:journalctl@ceph.mon.c.smithi002.stdout: RuntimeError: Failed command: systemctl restart ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem: Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:27:59.989 INFO:journalctl@ceph.mon.c.smithi002.stdout: See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:27:59.990 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:27:59 smithi002 ceph-mon[66732]: from='mon.1 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-09-17T00:27:59.990 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:27:59 smithi002 ceph-mon[66732]: Upgrade: unsafe to stop osd(s) at this time (13 PGs are or would become offline) 2024-09-17T00:27:59.990 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:27:59 smithi002 ceph-mon[66732]: pgmap v165: 161 pgs: 3 activating, 4 peering, 154 active+clean; 585 KiB data, 194 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:27:59.990 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:27:59 smithi002 ceph-mon[66732]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 118/726 objects degraded (16.253%), 24 pgs degraded) 2024-09-17T00:27:59.990 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:27:59 smithi002 ceph-mon[66732]: Cluster is now healthy 2024-09-17T00:27:59.990 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:27:59 smithi002 ceph-mon[63804]: pgmap v165: 161 pgs: 3 activating, 4 peering, 154 active+clean; 585 KiB data, 194 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:27:59.990 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:27:59 smithi002 ceph-mon[63804]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 118/726 objects degraded (16.253%), 24 pgs degraded) 2024-09-17T00:27:59.990 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:27:59 smithi002 ceph-mon[63804]: Cluster is now healthy 2024-09-17T00:28:01.308 INFO:teuthology.orchestra.run.smithi002.stdout:true 2024-09-17T00:28:01.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:28:01 smithi099 ceph-mon[56712]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 194 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:28:01.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:28:01 smithi002 ceph-mon[66732]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 194 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:28:01.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:28:01 smithi002 ceph-mon[63804]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 194 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:28:01.988 INFO:teuthology.orchestra.run.smithi002.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-17T00:28:01.988 INFO:teuthology.orchestra.run.smithi002.stdout:alertmanager.a smithi002 *:9093,9094 running (5m) 49s ago 10m 26.1M - 0.25.0 c8568f914cd2 a21e13b3d677 2024-09-17T00:28:01.989 INFO:teuthology.orchestra.run.smithi002.stdout:grafana.a smithi099 *:3000 running (5m) 11s ago 10m 53.6M - 8.3.5 dad864ee21e9 bafb824a3c08 2024-09-17T00:28:01.989 INFO:teuthology.orchestra.run.smithi002.stdout:iscsi.foo.smithi002.vmouem smithi002 running (69s) 49s ago 9m 48.3M - 3.5 e1d6a67b021e b5424a0496a5 2024-09-17T00:28:01.989 INFO:teuthology.orchestra.run.smithi002.stdout:mgr.x smithi099 *:8443,9283,8765 running (4m) 11s ago 15m 442M - 19.1.1-330-gc2ddf972 ebb0615149de 6aa2ccef9ca0 2024-09-17T00:28:01.989 INFO:teuthology.orchestra.run.smithi002.stdout:mgr.y smithi002 *:8443,9283,8765 running (5m) 49s ago 17m 506M - 19.1.1-330-gc2ddf972 ebb0615149de 8838897e8449 2024-09-17T00:28:01.989 INFO:teuthology.orchestra.run.smithi002.stdout:mon.a smithi002 running (4m) 49s ago 17m 51.3M 2048M 19.1.1-330-gc2ddf972 ebb0615149de a035def53ef6 2024-09-17T00:28:01.989 INFO:teuthology.orchestra.run.smithi002.stdout:mon.b smithi099 running (3m) 11s ago 15m 39.6M 2048M 19.1.1-330-gc2ddf972 ebb0615149de aab3ab73a850 2024-09-17T00:28:01.989 INFO:teuthology.orchestra.run.smithi002.stdout:mon.c smithi002 running (4m) 49s ago 15m 43.4M 2048M 19.1.1-330-gc2ddf972 ebb0615149de 8a510255ae66 2024-09-17T00:28:01.989 INFO:teuthology.orchestra.run.smithi002.stdout:node-exporter.a smithi002 *:9100 running (6m) 49s ago 11m 14.8M - 1.5.0 0da6a335fe13 1030ae01bb32 2024-09-17T00:28:01.989 INFO:teuthology.orchestra.run.smithi002.stdout:node-exporter.b smithi099 *:9100 running (5m) 11s ago 11m 16.5M - 1.5.0 0da6a335fe13 01fcd35dca8a 2024-09-17T00:28:01.989 INFO:teuthology.orchestra.run.smithi002.stdout:osd.0 smithi002 running (2m) 49s ago 15m 45.8M 2737M 19.1.1-330-gc2ddf972 ebb0615149de 8345719cb6f0 2024-09-17T00:28:01.989 INFO:teuthology.orchestra.run.smithi002.stdout:osd.1 smithi002 running (2m) 49s ago 14m 51.5M 2737M 19.1.1-330-gc2ddf972 ebb0615149de 3dbc23d57c53 2024-09-17T00:28:01.989 INFO:teuthology.orchestra.run.smithi002.stdout:osd.2 smithi002 running (93s) 49s ago 14m 49.3M 2737M 19.1.1-330-gc2ddf972 ebb0615149de d21401b6763b 2024-09-17T00:28:01.989 INFO:teuthology.orchestra.run.smithi002.stdout:osd.3 smithi002 running (54s) 49s ago 13m 12.2M 2737M 19.1.1-330-gc2ddf972 ebb0615149de 980f2a25d096 2024-09-17T00:28:01.990 INFO:teuthology.orchestra.run.smithi002.stdout:osd.4 smithi099 running (15s) 11s ago 13m 11.2M 2992M 19.1.1-330-gc2ddf972 ebb0615149de f644e5be4db8 2024-09-17T00:28:01.990 INFO:teuthology.orchestra.run.smithi002.stdout:osd.5 smithi099 running (13m) 11s ago 13m 59.7M 2992M 17.2.0 e1d6a67b021e 4f7ef3aefc58 2024-09-17T00:28:01.990 INFO:teuthology.orchestra.run.smithi002.stdout:osd.6 smithi099 running (12m) 11s ago 12m 54.9M 2992M 17.2.0 e1d6a67b021e 3213f5e6952a 2024-09-17T00:28:01.990 INFO:teuthology.orchestra.run.smithi002.stdout:osd.7 smithi099 running (12m) 11s ago 12m 62.5M 2992M 17.2.0 e1d6a67b021e 63be3a1ed0a1 2024-09-17T00:28:01.990 INFO:teuthology.orchestra.run.smithi002.stdout:prometheus.a smithi099 *:9095 running (4m) 11s ago 11m 67.2M - 2.43.0 a07b618ecd1d bf32c2ae11dc 2024-09-17T00:28:01.990 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.foo.smithi002.uwdhcs smithi002 *:8000 running (10m) 49s ago 10m 76.1M - 17.2.0 e1d6a67b021e 9c35813f459a 2024-09-17T00:28:01.990 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.foo.smithi099.utzfap smithi099 *:8000 running (10m) 11s ago 9m 76.8M - 17.2.0 e1d6a67b021e 34ec989160a6 2024-09-17T00:28:01.990 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.smpl.smithi002.rqnxii smithi002 *:80 running (9m) 49s ago 9m 74.4M - 17.2.0 e1d6a67b021e 0e180d0afb99 2024-09-17T00:28:01.990 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.smpl.smithi099.nzvkpm smithi099 *:80 running (9m) 11s ago 9m 77.3M - 17.2.0 e1d6a67b021e 6a068e82ac72 2024-09-17T00:28:02.409 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-09-17T00:28:02.409 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": { 2024-09-17T00:28:02.409 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 19.1.1-330-gc2ddf972 (c2ddf9721ef254645201cdbabd9c8f49c30a586f) squid (stable)": 3 2024-09-17T00:28:02.409 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-09-17T00:28:02.409 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": { 2024-09-17T00:28:02.409 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 19.1.1-330-gc2ddf972 (c2ddf9721ef254645201cdbabd9c8f49c30a586f) squid (stable)": 2 2024-09-17T00:28:02.410 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-09-17T00:28:02.410 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": { 2024-09-17T00:28:02.410 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3, 2024-09-17T00:28:02.410 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 19.1.1-330-gc2ddf972 (c2ddf9721ef254645201cdbabd9c8f49c30a586f) squid (stable)": 5 2024-09-17T00:28:02.410 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-09-17T00:28:02.410 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw": { 2024-09-17T00:28:02.410 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-09-17T00:28:02.410 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-09-17T00:28:02.410 INFO:teuthology.orchestra.run.smithi002.stdout: "overall": { 2024-09-17T00:28:02.410 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 7, 2024-09-17T00:28:02.410 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 19.1.1-330-gc2ddf972 (c2ddf9721ef254645201cdbabd9c8f49c30a586f) squid (stable)": 10 2024-09-17T00:28:02.410 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-09-17T00:28:02.410 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-09-17T00:28:02.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:28:02 smithi002 ceph-mon[66732]: from='client.34671 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:28:02.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:28:02 smithi002 ceph-mon[66732]: from='client.? 172.21.15.2:0/722864186' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:28:02.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:28:02 smithi002 ceph-mon[63804]: from='client.34671 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:28:02.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:28:02 smithi002 ceph-mon[63804]: from='client.? 172.21.15.2:0/722864186' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:28:02.802 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-09-17T00:28:02.802 INFO:teuthology.orchestra.run.smithi002.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f", 2024-09-17T00:28:02.802 INFO:teuthology.orchestra.run.smithi002.stdout: "in_progress": true, 2024-09-17T00:28:02.802 INFO:teuthology.orchestra.run.smithi002.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-17T00:28:02.802 INFO:teuthology.orchestra.run.smithi002.stdout: "services_complete": [ 2024-09-17T00:28:02.802 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr", 2024-09-17T00:28:02.802 INFO:teuthology.orchestra.run.smithi002.stdout: "mon" 2024-09-17T00:28:02.803 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-09-17T00:28:02.803 INFO:teuthology.orchestra.run.smithi002.stdout: "progress": "10/23 daemons upgraded", 2024-09-17T00:28:02.803 INFO:teuthology.orchestra.run.smithi002.stdout: "message": "Currently upgrading osd daemons", 2024-09-17T00:28:02.803 INFO:teuthology.orchestra.run.smithi002.stdout: "is_paused": false 2024-09-17T00:28:02.803 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-09-17T00:28:02.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:28:02 smithi099 ceph-mon[56712]: from='client.34671 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:28:02.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:28:02 smithi099 ceph-mon[56712]: from='client.? 172.21.15.2:0/722864186' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:28:03.215 INFO:teuthology.orchestra.run.smithi002.stdout:HEALTH_OK 2024-09-17T00:28:03.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:28:03 smithi099 ceph-mon[56712]: from='client.34677 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:28:03.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:28:03 smithi099 ceph-mon[56712]: from='client.44590 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:28:03.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:28:03 smithi099 ceph-mon[56712]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 194 MiB used, 715 GiB / 715 GiB avail; 16 KiB/s rd, 0 B/s wr, 25 op/s 2024-09-17T00:28:03.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:28:03 smithi099 ceph-mon[56712]: from='client.? 172.21.15.2:0/3116777628' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T00:28:03.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:28:03 smithi002 ceph-mon[66732]: from='client.34677 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:28:03.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:28:03 smithi002 ceph-mon[66732]: from='client.44590 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:28:03.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:28:03 smithi002 ceph-mon[66732]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 194 MiB used, 715 GiB / 715 GiB avail; 16 KiB/s rd, 0 B/s wr, 25 op/s 2024-09-17T00:28:03.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:28:03 smithi002 ceph-mon[66732]: from='client.? 172.21.15.2:0/3116777628' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T00:28:03.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:28:03 smithi002 ceph-mon[63804]: from='client.34677 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:28:03.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:28:03 smithi002 ceph-mon[63804]: from='client.44590 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:28:03.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:28:03 smithi002 ceph-mon[63804]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 194 MiB used, 715 GiB / 715 GiB avail; 16 KiB/s rd, 0 B/s wr, 25 op/s 2024-09-17T00:28:03.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:28:03 smithi002 ceph-mon[63804]: from='client.? 172.21.15.2:0/3116777628' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T00:28:04.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:28:04 smithi099 ceph-mon[56712]: from='client.44602 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:28:04.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:28:04 smithi002 ceph-mon[66732]: from='client.44602 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:28:04.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:28:04 smithi002 ceph-mon[63804]: from='client.44602 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:28:05.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:28:05 smithi099 ceph-mon[56712]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 195 MiB used, 715 GiB / 715 GiB avail; 66 KiB/s rd, 0 B/s wr, 110 op/s 2024-09-17T00:28:05.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:28:05 smithi002 ceph-mon[66732]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 195 MiB used, 715 GiB / 715 GiB avail; 66 KiB/s rd, 0 B/s wr, 110 op/s 2024-09-17T00:28:05.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:28:05 smithi002 ceph-mon[63804]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 195 MiB used, 715 GiB / 715 GiB avail; 66 KiB/s rd, 0 B/s wr, 110 op/s 2024-09-17T00:28:06.899 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:28:06 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: ::ffff:172.21.15.99 - - [17/Sep/2024:00:28:06] "GET /metrics HTTP/1.1" 200 38315 "" "Prometheus/2.43.0" 2024-09-17T00:28:07.605 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:28:07 smithi002 ceph-mon[63804]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 195 MiB used, 715 GiB / 715 GiB avail; 57 KiB/s rd, 0 B/s wr, 95 op/s 2024-09-17T00:28:07.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:28:07 smithi099 ceph-mon[56712]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 195 MiB used, 715 GiB / 715 GiB avail; 57 KiB/s rd, 0 B/s wr, 95 op/s 2024-09-17T00:28:07.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:28:07 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:28:07.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:28:07 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:28:07.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:28:07 smithi002 ceph-mon[66732]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 195 MiB used, 715 GiB / 715 GiB avail; 57 KiB/s rd, 0 B/s wr, 95 op/s 2024-09-17T00:28:07.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:28:07 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:28:09.875 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:28:09 smithi002 ceph-mon[63804]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 199 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-09-17T00:28:09.876 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:28:09 smithi002 ceph-mon[66732]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 199 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-09-17T00:28:09.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:28:09 smithi099 ceph-mon[56712]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 199 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-09-17T00:28:10.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:28:10 smithi099 ceph-mon[56712]: from='client.? 172.21.15.2:0/3726665710' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-17T00:28:10.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:28:10 smithi002 ceph-mon[66732]: from='client.? 172.21.15.2:0/3726665710' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-17T00:28:10.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:28:10 smithi002 ceph-mon[63804]: from='client.? 172.21.15.2:0/3726665710' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-17T00:28:11.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:28:11 smithi099 ceph-mon[56712]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 199 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-09-17T00:28:11.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:28:11 smithi002 ceph-mon[66732]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 199 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-09-17T00:28:11.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:28:11 smithi002 ceph-mon[63804]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 199 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-09-17T00:28:13.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:28:12 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:28:13.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:28:12 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T00:28:13.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:28:12 smithi099 ceph-mon[56712]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 199 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 119 op/s 2024-09-17T00:28:13.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:28:12 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:28:13.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:28:12 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T00:28:13.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:28:12 smithi002 ceph-mon[66732]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 199 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 119 op/s 2024-09-17T00:28:13.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:28:12 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:28:13.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:28:12 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T00:28:13.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:28:12 smithi002 ceph-mon[63804]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 199 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 119 op/s 2024-09-17T00:28:13.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:28:13 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-09-17T00:28:13.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:28:13 smithi099 ceph-mon[56712]: from='mon.1 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-09-17T00:28:13.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:28:13 smithi099 ceph-mon[56712]: Upgrade: osd.5 is safe to restart 2024-09-17T00:28:14.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:28:13 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-09-17T00:28:14.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:28:13 smithi002 ceph-mon[66732]: from='mon.1 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-09-17T00:28:14.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:28:13 smithi002 ceph-mon[66732]: Upgrade: osd.5 is safe to restart 2024-09-17T00:28:14.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:28:13 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-09-17T00:28:14.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:28:13 smithi002 ceph-mon[63804]: from='mon.1 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-09-17T00:28:14.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:28:13 smithi002 ceph-mon[63804]: Upgrade: osd.5 is safe to restart 2024-09-17T00:28:15.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:28:15 smithi099 ceph-mon[56712]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 199 MiB used, 715 GiB / 715 GiB avail; 67 KiB/s rd, 0 B/s wr, 112 op/s 2024-09-17T00:28:15.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:28:15 smithi099 ceph-mon[56712]: Upgrade: Updating osd.5 2024-09-17T00:28:15.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:28:15 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:28:15.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:28:15 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-09-17T00:28:15.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:28:15 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:28:15.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:28:15 smithi099 ceph-mon[56712]: Deploying daemon osd.5 on smithi099 2024-09-17T00:28:15.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:28:15 smithi002 ceph-mon[66732]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 199 MiB used, 715 GiB / 715 GiB avail; 67 KiB/s rd, 0 B/s wr, 112 op/s 2024-09-17T00:28:15.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:28:15 smithi002 ceph-mon[66732]: Upgrade: Updating osd.5 2024-09-17T00:28:15.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:28:15 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:28:15.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:28:15 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-09-17T00:28:15.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:28:15 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:28:15.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:28:15 smithi002 ceph-mon[66732]: Deploying daemon osd.5 on smithi099 2024-09-17T00:28:15.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:28:15 smithi002 ceph-mon[63804]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 199 MiB used, 715 GiB / 715 GiB avail; 67 KiB/s rd, 0 B/s wr, 112 op/s 2024-09-17T00:28:15.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:28:15 smithi002 ceph-mon[63804]: Upgrade: Updating osd.5 2024-09-17T00:28:15.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:28:15 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:28:15.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:28:15 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-09-17T00:28:15.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:28:15 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:28:15.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:28:15 smithi002 ceph-mon[63804]: Deploying daemon osd.5 on smithi099 2024-09-17T00:28:16.979 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:28:16 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: ::ffff:172.21.15.99 - - [17/Sep/2024:00:28:16] "GET /metrics HTTP/1.1" 200 38452 "" "Prometheus/2.43.0" 2024-09-17T00:28:17.677 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:28:17 smithi099 ceph-mon[56712]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 199 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 37 op/s 2024-09-17T00:28:17.677 INFO:journalctl@ceph.osd.5.smithi099.stdout:Sep 17 00:28:17 smithi099 systemd[1]: Stopping Ceph osd.5 for 3aad27a2-7489-11ef-bceb-c7b262605968... 2024-09-17T00:28:17.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:28:17 smithi002 ceph-mon[63804]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 199 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 37 op/s 2024-09-17T00:28:17.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:28:17 smithi002 ceph-mon[66732]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 199 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 37 op/s 2024-09-17T00:28:17.967 INFO:journalctl@ceph.osd.5.smithi099.stdout:Sep 17 00:28:17 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-5[30851]: 2024-09-17T00:28:17.788+0000 7f5caf8c6700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-17T00:28:17.967 INFO:journalctl@ceph.osd.5.smithi099.stdout:Sep 17 00:28:17 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-5[30851]: 2024-09-17T00:28:17.788+0000 7f5caf8c6700 -1 osd.5 137 *** Got signal Terminated *** 2024-09-17T00:28:17.967 INFO:journalctl@ceph.osd.5.smithi099.stdout:Sep 17 00:28:17 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-5[30851]: 2024-09-17T00:28:17.788+0000 7f5caf8c6700 -1 osd.5 137 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-17T00:28:18.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:28:18 smithi099 ceph-mon[56712]: osd.5 marked itself down and dead 2024-09-17T00:28:18.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:28:18 smithi002 ceph-mon[63804]: osd.5 marked itself down and dead 2024-09-17T00:28:18.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:28:18 smithi002 ceph-mon[66732]: osd.5 marked itself down and dead 2024-09-17T00:28:19.092 INFO:journalctl@ceph.osd.5.smithi099.stdout:Sep 17 00:28:18 smithi099 podman[61789]: 2024-09-17 00:28:18.838082242 +0000 UTC m=+1.235929222 container died 4f7ef3aefc5820202473b33a682c909d413b32d1f2a8e9a12cc6f87aa44fca47 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-5, ceph=True, io.openshift.tags=base centos centos-stream, io.k8s.display-name=CentOS Stream 8, GIT_BRANCH=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, maintainer=Guillaume Abrioux , GIT_CLEAN=True, architecture=x86_64, release=754, RELEASE=HEAD, io.openshift.expose-services=, vcs-type=git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vendor=Red Hat, Inc., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, version=8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.component=centos-stream-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.buildah.version=1.19.8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, CEPH_POINT_RELEASE=-17.2.0, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, name=centos-stream, build-date=2022-05-03T08:36:31.336870, GIT_REPO=https://github.com/ceph/ceph-container.git, distribution-scope=public) 2024-09-17T00:28:19.396 INFO:journalctl@ceph.osd.5.smithi099.stdout:Sep 17 00:28:19 smithi099 podman[61789]: 2024-09-17 00:28:19.194095131 +0000 UTC m=+1.591942102 container cleanup 4f7ef3aefc5820202473b33a682c909d413b32d1f2a8e9a12cc6f87aa44fca47 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-5, vendor=Red Hat, Inc., ceph=True, architecture=x86_64, vcs-type=git, CEPH_POINT_RELEASE=-17.2.0, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_CLEAN=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.display-name=CentOS Stream 8, distribution-scope=public, build-date=2022-05-03T08:36:31.336870, release=754, io.openshift.expose-services=, GIT_REPO=https://github.com/ceph/ceph-container.git, version=8, io.openshift.tags=base centos centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=centos-stream-container, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, RELEASE=HEAD, name=centos-stream, GIT_BRANCH=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Guillaume Abrioux , io.buildah.version=1.19.8) 2024-09-17T00:28:19.397 INFO:journalctl@ceph.osd.5.smithi099.stdout:Sep 17 00:28:19 smithi099 bash[61789]: ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-5 2024-09-17T00:28:19.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:28:19 smithi002 ceph-mon[63804]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 216 MiB used, 715 GiB / 715 GiB avail; 77 KiB/s rd, 0 B/s wr, 127 op/s 2024-09-17T00:28:19.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:28:19 smithi002 ceph-mon[63804]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-17T00:28:19.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:28:19 smithi002 ceph-mon[63804]: osdmap e138: 8 total, 7 up, 8 in 2024-09-17T00:28:19.480 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:28:19 smithi002 ceph-mon[66732]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 216 MiB used, 715 GiB / 715 GiB avail; 77 KiB/s rd, 0 B/s wr, 127 op/s 2024-09-17T00:28:19.480 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:28:19 smithi002 ceph-mon[66732]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-17T00:28:19.480 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:28:19 smithi002 ceph-mon[66732]: osdmap e138: 8 total, 7 up, 8 in 2024-09-17T00:28:19.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:28:19 smithi099 ceph-mon[56712]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 216 MiB used, 715 GiB / 715 GiB avail; 77 KiB/s rd, 0 B/s wr, 127 op/s 2024-09-17T00:28:19.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:28:19 smithi099 ceph-mon[56712]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-17T00:28:19.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:28:19 smithi099 ceph-mon[56712]: osdmap e138: 8 total, 7 up, 8 in 2024-09-17T00:28:19.718 INFO:journalctl@ceph.osd.5.smithi099.stdout:Sep 17 00:28:19 smithi099 podman[61814]: 2024-09-17 00:28:19.504226494 +0000 UTC m=+0.667281145 container remove 4f7ef3aefc5820202473b33a682c909d413b32d1f2a8e9a12cc6f87aa44fca47 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-5, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, CEPH_POINT_RELEASE=-17.2.0, com.redhat.component=centos-stream-container, io.openshift.expose-services=, GIT_CLEAN=True, release=754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_BRANCH=HEAD, ceph=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, build-date=2022-05-03T08:36:31.336870, version=8, io.openshift.tags=base centos centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.buildah.version=1.19.8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, distribution-scope=public, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_REPO=https://github.com/ceph/ceph-container.git, vendor=Red Hat, Inc., io.k8s.display-name=CentOS Stream 8, name=centos-stream, vcs-type=git, maintainer=Guillaume Abrioux , architecture=x86_64, RELEASE=HEAD) 2024-09-17T00:28:20.081 INFO:journalctl@ceph.osd.5.smithi099.stdout:Sep 17 00:28:19 smithi099 podman[61908]: 2024-09-17 00:28:19.71758357 +0000 UTC m=+0.015989266 image pull ebb0615149def4a3179c82c0cc47af75808300abb9f1142788e5e605a7ac0e5b quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f 2024-09-17T00:28:20.081 INFO:journalctl@ceph.osd.5.smithi099.stdout:Sep 17 00:28:19 smithi099 podman[61908]: 2024-09-17 00:28:19.855409367 +0000 UTC m=+0.153815090 container create 5a55be998c174cc8fcb5d01130467cfd6d9d1ad356f0b010455bae4e51ca9dc8 (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-5-deactivate, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, ceph=True, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, RELEASE=squid-c2ddf97) 2024-09-17T00:28:20.407 INFO:journalctl@ceph.osd.5.smithi099.stdout:Sep 17 00:28:20 smithi099 podman[61908]: 2024-09-17 00:28:20.083922275 +0000 UTC m=+0.382327965 container init 5a55be998c174cc8fcb5d01130467cfd6d9d1ad356f0b010455bae4e51ca9dc8 (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-5-deactivate, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-c2ddf97, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-09-17T00:28:20.407 INFO:journalctl@ceph.osd.5.smithi099.stdout:Sep 17 00:28:20 smithi099 podman[61908]: 2024-09-17 00:28:20.087420044 +0000 UTC m=+0.385825733 container start 5a55be998c174cc8fcb5d01130467cfd6d9d1ad356f0b010455bae4e51ca9dc8 (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-5-deactivate, RELEASE=squid-c2ddf97, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-17T00:28:20.407 INFO:journalctl@ceph.osd.5.smithi099.stdout:Sep 17 00:28:20 smithi099 podman[61908]: 2024-09-17 00:28:20.175501197 +0000 UTC m=+0.473906891 container attach 5a55be998c174cc8fcb5d01130467cfd6d9d1ad356f0b010455bae4e51ca9dc8 (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-5-deactivate, io.buildah.version=1.37.2, RELEASE=squid-c2ddf97, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-17T00:28:20.407 INFO:journalctl@ceph.osd.5.smithi099.stdout:Sep 17 00:28:20 smithi099 podman[61908]: 2024-09-17 00:28:20.273941834 +0000 UTC m=+0.572347515 container died 5a55be998c174cc8fcb5d01130467cfd6d9d1ad356f0b010455bae4e51ca9dc8 (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-5-deactivate, RELEASE=squid-c2ddf97, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-09-17T00:28:20.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:28:20 smithi099 ceph-mon[56712]: osdmap e139: 8 total, 7 up, 8 in 2024-09-17T00:28:20.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:28:20 smithi099 ceph-mon[56712]: from='client.44626 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:28:20.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:28:20 smithi002 ceph-mon[63804]: osdmap e139: 8 total, 7 up, 8 in 2024-09-17T00:28:20.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:28:20 smithi002 ceph-mon[63804]: from='client.44626 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:28:20.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:28:20 smithi002 ceph-mon[66732]: osdmap e139: 8 total, 7 up, 8 in 2024-09-17T00:28:20.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:28:20 smithi002 ceph-mon[66732]: from='client.44626 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:28:21.217 INFO:journalctl@ceph.osd.5.smithi099.stdout:Sep 17 00:28:20 smithi099 podman[61931]: 2024-09-17 00:28:20.8655359 +0000 UTC m=+0.585945709 container remove 5a55be998c174cc8fcb5d01130467cfd6d9d1ad356f0b010455bae4e51ca9dc8 (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-5-deactivate, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=squid-c2ddf97, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.37.2) 2024-09-17T00:28:21.217 INFO:journalctl@ceph.osd.5.smithi099.stdout:Sep 17 00:28:20 smithi099 systemd[1]: ceph-3aad27a2-7489-11ef-bceb-c7b262605968@osd.5.service: Deactivated successfully. 2024-09-17T00:28:21.217 INFO:journalctl@ceph.osd.5.smithi099.stdout:Sep 17 00:28:20 smithi099 systemd[1]: Stopped Ceph osd.5 for 3aad27a2-7489-11ef-bceb-c7b262605968. 2024-09-17T00:28:21.217 INFO:journalctl@ceph.osd.5.smithi099.stdout:Sep 17 00:28:20 smithi099 systemd[1]: ceph-3aad27a2-7489-11ef-bceb-c7b262605968@osd.5.service: Consumed 9.313s CPU time. 2024-09-17T00:28:21.217 INFO:journalctl@ceph.osd.5.smithi099.stdout:Sep 17 00:28:21 smithi099 systemd[1]: Starting Ceph osd.5 for 3aad27a2-7489-11ef-bceb-c7b262605968... 2024-09-17T00:28:21.609 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:28:21 smithi099 ceph-mon[56712]: pgmap v178: 161 pgs: 6 active+undersized, 10 peering, 15 stale+active+clean, 130 active+clean; 585 KiB data, 220 MiB used, 715 GiB / 715 GiB avail; 97 KiB/s rd, 0 B/s wr, 162 op/s 2024-09-17T00:28:21.609 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:28:21 smithi099 ceph-mon[56712]: Health check failed: Reduced data availability: 1 pg inactive, 2 pgs peering (PG_AVAILABILITY) 2024-09-17T00:28:21.609 INFO:journalctl@ceph.osd.5.smithi099.stdout:Sep 17 00:28:21 smithi099 podman[62031]: 2024-09-17 00:28:21.207988354 +0000 UTC m=+0.015937598 image pull ebb0615149def4a3179c82c0cc47af75808300abb9f1142788e5e605a7ac0e5b quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f 2024-09-17T00:28:21.609 INFO:journalctl@ceph.osd.5.smithi099.stdout:Sep 17 00:28:21 smithi099 podman[62031]: 2024-09-17 00:28:21.326466054 +0000 UTC m=+0.134415282 container create 8e0d6cc8256f3cbf6da975c8b3a35c622b2befd966b1432be55d7a3aab85e386 (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-5-activate, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, RELEASE=squid-c2ddf97, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-17T00:28:21.609 INFO:journalctl@ceph.osd.5.smithi099.stdout:Sep 17 00:28:21 smithi099 podman[62031]: 2024-09-17 00:28:21.537516916 +0000 UTC m=+0.345466159 container init 8e0d6cc8256f3cbf6da975c8b3a35c622b2befd966b1432be55d7a3aab85e386 (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-5-activate, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, maintainer=Guillaume Abrioux , RELEASE=squid-c2ddf97, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-09-17T00:28:21.609 INFO:journalctl@ceph.osd.5.smithi099.stdout:Sep 17 00:28:21 smithi099 podman[62031]: 2024-09-17 00:28:21.541051067 +0000 UTC m=+0.349000301 container start 8e0d6cc8256f3cbf6da975c8b3a35c622b2befd966b1432be55d7a3aab85e386 (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-5-activate, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-c2ddf97, org.label-schema.build-date=20240731, GIT_CLEAN=True, ceph=True) 2024-09-17T00:28:21.610 INFO:journalctl@ceph.osd.5.smithi099.stdout:Sep 17 00:28:21 smithi099 podman[62031]: 2024-09-17 00:28:21.612339655 +0000 UTC m=+0.420288898 container attach 8e0d6cc8256f3cbf6da975c8b3a35c622b2befd966b1432be55d7a3aab85e386 (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-5-activate, ceph=True, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, RELEASE=squid-c2ddf97, io.buildah.version=1.37.2, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-09-17T00:28:21.732 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:28:21 smithi002 ceph-mon[63804]: pgmap v178: 161 pgs: 6 active+undersized, 10 peering, 15 stale+active+clean, 130 active+clean; 585 KiB data, 220 MiB used, 715 GiB / 715 GiB avail; 97 KiB/s rd, 0 B/s wr, 162 op/s 2024-09-17T00:28:21.732 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:28:21 smithi002 ceph-mon[63804]: Health check failed: Reduced data availability: 1 pg inactive, 2 pgs peering (PG_AVAILABILITY) 2024-09-17T00:28:21.733 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:28:21 smithi002 ceph-mon[66732]: pgmap v178: 161 pgs: 6 active+undersized, 10 peering, 15 stale+active+clean, 130 active+clean; 585 KiB data, 220 MiB used, 715 GiB / 715 GiB avail; 97 KiB/s rd, 0 B/s wr, 162 op/s 2024-09-17T00:28:21.733 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:28:21 smithi002 ceph-mon[66732]: Health check failed: Reduced data availability: 1 pg inactive, 2 pgs peering (PG_AVAILABILITY) 2024-09-17T00:28:21.967 INFO:journalctl@ceph.osd.5.smithi099.stdout:Sep 17 00:28:21 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-5-activate[62045]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-17T00:28:21.967 INFO:journalctl@ceph.osd.5.smithi099.stdout:Sep 17 00:28:21 smithi099 bash[62031]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-17T00:28:21.967 INFO:journalctl@ceph.osd.5.smithi099.stdout:Sep 17 00:28:21 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-5-activate[62045]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-17T00:28:21.967 INFO:journalctl@ceph.osd.5.smithi099.stdout:Sep 17 00:28:21 smithi099 bash[62031]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-17T00:28:22.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:28:22 smithi099 ceph-mon[56712]: Health check failed: Degraded data redundancy: 25/726 objects degraded (3.444%), 6 pgs degraded (PG_DEGRADED) 2024-09-17T00:28:22.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:28:22 smithi002 ceph-mon[66732]: Health check failed: Degraded data redundancy: 25/726 objects degraded (3.444%), 6 pgs degraded (PG_DEGRADED) 2024-09-17T00:28:22.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:28:22 smithi002 ceph-mon[63804]: Health check failed: Degraded data redundancy: 25/726 objects degraded (3.444%), 6 pgs degraded (PG_DEGRADED) 2024-09-17T00:28:23.038 INFO:journalctl@ceph.osd.5.smithi099.stdout:Sep 17 00:28:22 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-5-activate[62045]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-09-17T00:28:23.038 INFO:journalctl@ceph.osd.5.smithi099.stdout:Sep 17 00:28:22 smithi099 bash[62031]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-09-17T00:28:23.038 INFO:journalctl@ceph.osd.5.smithi099.stdout:Sep 17 00:28:22 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-5-activate[62045]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-09-17T00:28:23.038 INFO:journalctl@ceph.osd.5.smithi099.stdout:Sep 17 00:28:22 smithi099 bash[62031]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-09-17T00:28:23.038 INFO:journalctl@ceph.osd.5.smithi099.stdout:Sep 17 00:28:23 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-5-activate[62045]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-09-17T00:28:23.038 INFO:journalctl@ceph.osd.5.smithi099.stdout:Sep 17 00:28:23 smithi099 bash[62031]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-09-17T00:28:23.355 INFO:journalctl@ceph.osd.5.smithi099.stdout:Sep 17 00:28:23 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-5-activate[62045]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-09-17T00:28:23.355 INFO:journalctl@ceph.osd.5.smithi099.stdout:Sep 17 00:28:23 smithi099 bash[62031]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-09-17T00:28:23.355 INFO:journalctl@ceph.osd.5.smithi099.stdout:Sep 17 00:28:23 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-5-activate[62045]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-5/block 2024-09-17T00:28:23.355 INFO:journalctl@ceph.osd.5.smithi099.stdout:Sep 17 00:28:23 smithi099 bash[62031]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-5/block 2024-09-17T00:28:23.355 INFO:journalctl@ceph.osd.5.smithi099.stdout:Sep 17 00:28:23 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-5-activate[62045]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-09-17T00:28:23.356 INFO:journalctl@ceph.osd.5.smithi099.stdout:Sep 17 00:28:23 smithi099 bash[62031]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-09-17T00:28:23.356 INFO:journalctl@ceph.osd.5.smithi099.stdout:Sep 17 00:28:23 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-5-activate[62045]: --> ceph-volume raw activate successful for osd ID: 5 2024-09-17T00:28:23.356 INFO:journalctl@ceph.osd.5.smithi099.stdout:Sep 17 00:28:23 smithi099 bash[62031]: --> ceph-volume raw activate successful for osd ID: 5 2024-09-17T00:28:23.356 INFO:journalctl@ceph.osd.5.smithi099.stdout:Sep 17 00:28:23 smithi099 podman[62031]: 2024-09-17 00:28:23.069157836 +0000 UTC m=+1.877107069 container died 8e0d6cc8256f3cbf6da975c8b3a35c622b2befd966b1432be55d7a3aab85e386 (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-5-activate, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=squid-c2ddf97, org.label-schema.build-date=20240731, GIT_CLEAN=True, ceph=True) 2024-09-17T00:28:23.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:28:23 smithi099 ceph-mon[56712]: pgmap v179: 161 pgs: 19 active+undersized, 10 peering, 9 stale+active+clean, 6 active+undersized+degraded, 117 active+clean; 585 KiB data, 220 MiB used, 715 GiB / 715 GiB avail; 93 KiB/s rd, 0 B/s wr, 158 op/s; 25/726 objects degraded (3.444%) 2024-09-17T00:28:23.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:28:23 smithi002 ceph-mon[66732]: pgmap v179: 161 pgs: 19 active+undersized, 10 peering, 9 stale+active+clean, 6 active+undersized+degraded, 117 active+clean; 585 KiB data, 220 MiB used, 715 GiB / 715 GiB avail; 93 KiB/s rd, 0 B/s wr, 158 op/s; 25/726 objects degraded (3.444%) 2024-09-17T00:28:23.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:28:23 smithi002 ceph-mon[63804]: pgmap v179: 161 pgs: 19 active+undersized, 10 peering, 9 stale+active+clean, 6 active+undersized+degraded, 117 active+clean; 585 KiB data, 220 MiB used, 715 GiB / 715 GiB avail; 93 KiB/s rd, 0 B/s wr, 158 op/s; 25/726 objects degraded (3.444%) 2024-09-17T00:28:24.097 INFO:journalctl@ceph.osd.5.smithi099.stdout:Sep 17 00:28:23 smithi099 podman[62188]: 2024-09-17 00:28:23.783818617 +0000 UTC m=+0.708468885 container remove 8e0d6cc8256f3cbf6da975c8b3a35c622b2befd966b1432be55d7a3aab85e386 (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-5-activate, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, RELEASE=squid-c2ddf97, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS) 2024-09-17T00:28:24.097 INFO:journalctl@ceph.osd.5.smithi099.stdout:Sep 17 00:28:23 smithi099 podman[62245]: 2024-09-17 00:28:23.893948072 +0000 UTC m=+0.015856162 image pull ebb0615149def4a3179c82c0cc47af75808300abb9f1142788e5e605a7ac0e5b quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f 2024-09-17T00:28:24.097 INFO:journalctl@ceph.osd.5.smithi099.stdout:Sep 17 00:28:24 smithi099 podman[62245]: 2024-09-17 00:28:24.001481674 +0000 UTC m=+0.123389748 container create 2df0d88d16ce71ff763e894d8734cd09ebe3c12c64772f073d54151a0df05394 (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-5, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , RELEASE=squid-c2ddf97, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=) 2024-09-17T00:28:24.717 INFO:journalctl@ceph.osd.5.smithi099.stdout:Sep 17 00:28:24 smithi099 podman[62245]: 2024-09-17 00:28:24.237946968 +0000 UTC m=+0.359855051 container init 2df0d88d16ce71ff763e894d8734cd09ebe3c12c64772f073d54151a0df05394 (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-5, maintainer=Guillaume Abrioux , RELEASE=squid-c2ddf97, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2) 2024-09-17T00:28:24.717 INFO:journalctl@ceph.osd.5.smithi099.stdout:Sep 17 00:28:24 smithi099 podman[62245]: 2024-09-17 00:28:24.24120255 +0000 UTC m=+0.363110635 container start 2df0d88d16ce71ff763e894d8734cd09ebe3c12c64772f073d54151a0df05394 (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-5, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-c2ddf97) 2024-09-17T00:28:24.717 INFO:journalctl@ceph.osd.5.smithi099.stdout:Sep 17 00:28:24 smithi099 ceph-osd[62264]: -- 172.21.15.99:0/3155203706 <== mon.2 v2:172.21.15.99:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55d74c11fa40 con 0x55d74c0fdc00 2024-09-17T00:28:24.717 INFO:journalctl@ceph.osd.5.smithi099.stdout:Sep 17 00:28:24 smithi099 bash[62245]: 2df0d88d16ce71ff763e894d8734cd09ebe3c12c64772f073d54151a0df05394 2024-09-17T00:28:24.717 INFO:journalctl@ceph.osd.5.smithi099.stdout:Sep 17 00:28:24 smithi099 systemd[1]: Started Ceph osd.5 for 3aad27a2-7489-11ef-bceb-c7b262605968. 2024-09-17T00:28:25.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:28:25 smithi099 ceph-mon[56712]: pgmap v180: 161 pgs: 39 active+undersized, 20 active+undersized+degraded, 102 active+clean; 585 KiB data, 220 MiB used, 715 GiB / 715 GiB avail; 92 KiB/s rd, 0 B/s wr, 157 op/s; 79/726 objects degraded (10.882%) 2024-09-17T00:28:25.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:28:25 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:28:25.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:28:25 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:28:25.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:28:25 smithi099 ceph-mon[56712]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 2 pgs peering) 2024-09-17T00:28:25.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:28:25 smithi002 ceph-mon[63804]: pgmap v180: 161 pgs: 39 active+undersized, 20 active+undersized+degraded, 102 active+clean; 585 KiB data, 220 MiB used, 715 GiB / 715 GiB avail; 92 KiB/s rd, 0 B/s wr, 157 op/s; 79/726 objects degraded (10.882%) 2024-09-17T00:28:25.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:28:25 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:28:25.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:28:25 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:28:25.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:28:25 smithi002 ceph-mon[63804]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 2 pgs peering) 2024-09-17T00:28:25.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:28:25 smithi002 ceph-mon[66732]: pgmap v180: 161 pgs: 39 active+undersized, 20 active+undersized+degraded, 102 active+clean; 585 KiB data, 220 MiB used, 715 GiB / 715 GiB avail; 92 KiB/s rd, 0 B/s wr, 157 op/s; 79/726 objects degraded (10.882%) 2024-09-17T00:28:25.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:28:25 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:28:25.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:28:25 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:28:25.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:28:25 smithi002 ceph-mon[66732]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 2 pgs peering) 2024-09-17T00:28:26.217 INFO:journalctl@ceph.osd.5.smithi099.stdout:Sep 17 00:28:25 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-5[62260]: 2024-09-17T00:28:25.902+0000 7f18799e2640 -1 Falling back to public interface 2024-09-17T00:28:26.979 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:28:26 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: ::ffff:172.21.15.99 - - [17/Sep/2024:00:28:26] "GET /metrics HTTP/1.1" 200 38452 "" "Prometheus/2.43.0" 2024-09-17T00:28:27.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:28:27 smithi099 ceph-mon[56712]: pgmap v181: 161 pgs: 39 active+undersized, 20 active+undersized+degraded, 102 active+clean; 585 KiB data, 220 MiB used, 715 GiB / 715 GiB avail; 11 KiB/s rd, 0 B/s wr, 22 op/s; 79/726 objects degraded (10.882%) 2024-09-17T00:28:27.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:28:27 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:28:27.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:28:27 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T00:28:27.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:28:27 smithi002 ceph-mon[63804]: pgmap v181: 161 pgs: 39 active+undersized, 20 active+undersized+degraded, 102 active+clean; 585 KiB data, 220 MiB used, 715 GiB / 715 GiB avail; 11 KiB/s rd, 0 B/s wr, 22 op/s; 79/726 objects degraded (10.882%) 2024-09-17T00:28:27.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:28:27 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:28:27.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:28:27 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T00:28:27.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:28:27 smithi002 ceph-mon[66732]: pgmap v181: 161 pgs: 39 active+undersized, 20 active+undersized+degraded, 102 active+clean; 585 KiB data, 220 MiB used, 715 GiB / 715 GiB avail; 11 KiB/s rd, 0 B/s wr, 22 op/s; 79/726 objects degraded (10.882%) 2024-09-17T00:28:27.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:28:27 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:28:27.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:28:27 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T00:28:29.683 INFO:journalctl@ceph.osd.5.smithi099.stdout:Sep 17 00:28:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-5[62260]: 2024-09-17T00:28:29.426+0000 7f18799e2640 -1 osd.5 0 read_superblock omap replica is missing. 2024-09-17T00:28:29.684 INFO:journalctl@ceph.osd.5.smithi099.stdout:Sep 17 00:28:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-5[62260]: 2024-09-17T00:28:29.456+0000 7f18799e2640 -1 osd.5 137 log_to_monitors true 2024-09-17T00:28:29.684 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:28:29 smithi099 ceph-mon[56712]: pgmap v182: 161 pgs: 39 active+undersized, 20 active+undersized+degraded, 102 active+clean; 585 KiB data, 220 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 0 B/s wr, 18 op/s; 79/726 objects degraded (10.882%) 2024-09-17T00:28:29.684 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:28:29 smithi099 ceph-mon[56712]: Health check update: Degraded data redundancy: 79/726 objects degraded (10.882%), 20 pgs degraded (PG_DEGRADED) 2024-09-17T00:28:29.684 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:28:29 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:28:29.684 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:28:29 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:28:29.684 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:28:29 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:28:29.684 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:28:29 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:28:29.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:28:29 smithi002 ceph-mon[66732]: pgmap v182: 161 pgs: 39 active+undersized, 20 active+undersized+degraded, 102 active+clean; 585 KiB data, 220 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 0 B/s wr, 18 op/s; 79/726 objects degraded (10.882%) 2024-09-17T00:28:29.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:28:29 smithi002 ceph-mon[66732]: Health check update: Degraded data redundancy: 79/726 objects degraded (10.882%), 20 pgs degraded (PG_DEGRADED) 2024-09-17T00:28:29.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:28:29 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:28:29.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:28:29 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:28:29.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:28:29 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:28:29.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:28:29 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:28:29.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:28:29 smithi002 ceph-mon[63804]: pgmap v182: 161 pgs: 39 active+undersized, 20 active+undersized+degraded, 102 active+clean; 585 KiB data, 220 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 0 B/s wr, 18 op/s; 79/726 objects degraded (10.882%) 2024-09-17T00:28:29.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:28:29 smithi002 ceph-mon[63804]: Health check update: Degraded data redundancy: 79/726 objects degraded (10.882%), 20 pgs degraded (PG_DEGRADED) 2024-09-17T00:28:29.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:28:29 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:28:29.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:28:29 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:28:29.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:28:29 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:28:29.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:28:29 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:28:30.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:28:30 smithi099 ceph-mon[56712]: from='osd.5 [v2:172.21.15.99:6808/4137663302,v1:172.21.15.99:6809/4137663302]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-17T00:28:30.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:28:30 smithi099 ceph-mon[56712]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-17T00:28:30.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:28:30 smithi099 ceph-mon[56712]: from='client.44626 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:28:30.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:28:30 smithi002 ceph-mon[63804]: from='osd.5 [v2:172.21.15.99:6808/4137663302,v1:172.21.15.99:6809/4137663302]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-17T00:28:30.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:28:30 smithi002 ceph-mon[63804]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-17T00:28:30.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:28:30 smithi002 ceph-mon[63804]: from='client.44626 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:28:30.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:28:30 smithi002 ceph-mon[66732]: from='osd.5 [v2:172.21.15.99:6808/4137663302,v1:172.21.15.99:6809/4137663302]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-17T00:28:30.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:28:30 smithi002 ceph-mon[66732]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-17T00:28:30.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:28:30 smithi002 ceph-mon[66732]: from='client.44626 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:28:31.086 INFO:journalctl@ceph.osd.5.smithi099.stdout:Sep 17 00:28:30 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-5[62260]: 2024-09-17T00:28:30.872+0000 7f186d785640 -1 osd.5 137 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T00:28:31.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:28:31 smithi099 ceph-mon[56712]: pgmap v183: 161 pgs: 39 active+undersized, 20 active+undersized+degraded, 102 active+clean; 585 KiB data, 220 MiB used, 715 GiB / 715 GiB avail; 3.8 KiB/s rd, 0 B/s wr, 8 op/s; 79/726 objects degraded (10.882%) 2024-09-17T00:28:31.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:28:31 smithi099 ceph-mon[56712]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-17T00:28:31.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:28:31 smithi099 ceph-mon[56712]: osdmap e140: 8 total, 7 up, 8 in 2024-09-17T00:28:31.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:28:31 smithi099 ceph-mon[56712]: from='osd.5 [v2:172.21.15.99:6808/4137663302,v1:172.21.15.99:6809/4137663302]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi099", "root=default"]}]: dispatch 2024-09-17T00:28:31.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:28:31 smithi099 ceph-mon[56712]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi099", "root=default"]}]: dispatch 2024-09-17T00:28:31.718 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:28:31 smithi099 ceph-mon[56712]: from='osd.5 ' entity='osd.5' 2024-09-17T00:28:31.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:28:31 smithi002 ceph-mon[63804]: pgmap v183: 161 pgs: 39 active+undersized, 20 active+undersized+degraded, 102 active+clean; 585 KiB data, 220 MiB used, 715 GiB / 715 GiB avail; 3.8 KiB/s rd, 0 B/s wr, 8 op/s; 79/726 objects degraded (10.882%) 2024-09-17T00:28:31.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:28:31 smithi002 ceph-mon[63804]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-17T00:28:31.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:28:31 smithi002 ceph-mon[63804]: osdmap e140: 8 total, 7 up, 8 in 2024-09-17T00:28:31.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:28:31 smithi002 ceph-mon[63804]: from='osd.5 [v2:172.21.15.99:6808/4137663302,v1:172.21.15.99:6809/4137663302]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi099", "root=default"]}]: dispatch 2024-09-17T00:28:31.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:28:31 smithi002 ceph-mon[63804]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi099", "root=default"]}]: dispatch 2024-09-17T00:28:31.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:28:31 smithi002 ceph-mon[63804]: from='osd.5 ' entity='osd.5' 2024-09-17T00:28:31.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:28:31 smithi002 ceph-mon[66732]: pgmap v183: 161 pgs: 39 active+undersized, 20 active+undersized+degraded, 102 active+clean; 585 KiB data, 220 MiB used, 715 GiB / 715 GiB avail; 3.8 KiB/s rd, 0 B/s wr, 8 op/s; 79/726 objects degraded (10.882%) 2024-09-17T00:28:31.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:28:31 smithi002 ceph-mon[66732]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-17T00:28:31.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:28:31 smithi002 ceph-mon[66732]: osdmap e140: 8 total, 7 up, 8 in 2024-09-17T00:28:31.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:28:31 smithi002 ceph-mon[66732]: from='osd.5 [v2:172.21.15.99:6808/4137663302,v1:172.21.15.99:6809/4137663302]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi099", "root=default"]}]: dispatch 2024-09-17T00:28:31.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:28:31 smithi002 ceph-mon[66732]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi099", "root=default"]}]: dispatch 2024-09-17T00:28:31.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:28:31 smithi002 ceph-mon[66732]: from='osd.5 ' entity='osd.5' 2024-09-17T00:28:32.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:28:32 smithi099 ceph-mon[56712]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-17T00:28:32.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:28:32 smithi099 ceph-mon[56712]: osd.5 [v2:172.21.15.99:6808/4137663302,v1:172.21.15.99:6809/4137663302] boot 2024-09-17T00:28:32.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:28:32 smithi099 ceph-mon[56712]: osdmap e141: 8 total, 8 up, 8 in 2024-09-17T00:28:32.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:28:32 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T00:28:32.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:28:32 smithi002 ceph-mon[63804]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-17T00:28:32.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:28:32 smithi002 ceph-mon[63804]: osd.5 [v2:172.21.15.99:6808/4137663302,v1:172.21.15.99:6809/4137663302] boot 2024-09-17T00:28:32.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:28:32 smithi002 ceph-mon[63804]: osdmap e141: 8 total, 8 up, 8 in 2024-09-17T00:28:32.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:28:32 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T00:28:32.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:28:32 smithi002 ceph-mon[66732]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-17T00:28:32.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:28:32 smithi002 ceph-mon[66732]: osd.5 [v2:172.21.15.99:6808/4137663302,v1:172.21.15.99:6809/4137663302] boot 2024-09-17T00:28:32.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:28:32 smithi002 ceph-mon[66732]: osdmap e141: 8 total, 8 up, 8 in 2024-09-17T00:28:32.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:28:32 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-17T00:28:33.586 INFO:teuthology.orchestra.run.smithi002.stdout:true 2024-09-17T00:28:33.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:28:33 smithi099 ceph-mon[56712]: pgmap v186: 161 pgs: 39 active+undersized, 20 active+undersized+degraded, 102 active+clean; 585 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 79/726 objects degraded (10.882%) 2024-09-17T00:28:33.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:28:33 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:28:33.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:28:33 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:28:33.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:28:33 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:28:33.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:28:33 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:28:33.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:28:33 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:28:33.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:28:33 smithi099 ceph-mon[56712]: Reconfiguring iscsi.foo.smithi002.vmouem (dependencies changed)... 2024-09-17T00:28:33.718 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:28:33 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi002.vmouem", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-17T00:28:33.718 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:28:33 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi002.vmouem", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-17T00:28:33.718 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:28:33 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:28:33.718 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:28:33 smithi099 ceph-mon[56712]: Reconfiguring daemon iscsi.foo.smithi002.vmouem on smithi002 2024-09-17T00:28:33.718 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:28:33 smithi099 ceph-mon[56712]: osdmap e142: 8 total, 8 up, 8 in 2024-09-17T00:28:33.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:28:33 smithi002 ceph-mon[66732]: pgmap v186: 161 pgs: 39 active+undersized, 20 active+undersized+degraded, 102 active+clean; 585 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 79/726 objects degraded (10.882%) 2024-09-17T00:28:33.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:28:33 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:28:33.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:28:33 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:28:33.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:28:33 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:28:33.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:28:33 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:28:33.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:28:33 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:28:33.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:28:33 smithi002 ceph-mon[66732]: Reconfiguring iscsi.foo.smithi002.vmouem (dependencies changed)... 2024-09-17T00:28:33.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:28:33 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi002.vmouem", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-17T00:28:33.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:28:33 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi002.vmouem", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-17T00:28:33.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:28:33 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:28:33.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:28:33 smithi002 ceph-mon[66732]: Reconfiguring daemon iscsi.foo.smithi002.vmouem on smithi002 2024-09-17T00:28:33.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:28:33 smithi002 ceph-mon[66732]: osdmap e142: 8 total, 8 up, 8 in 2024-09-17T00:28:33.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:28:33 smithi002 ceph-mon[63804]: pgmap v186: 161 pgs: 39 active+undersized, 20 active+undersized+degraded, 102 active+clean; 585 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 79/726 objects degraded (10.882%) 2024-09-17T00:28:33.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:28:33 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:28:33.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:28:33 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:28:33.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:28:33 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:28:33.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:28:33 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:28:33.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:28:33 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:28:33.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:28:33 smithi002 ceph-mon[63804]: Reconfiguring iscsi.foo.smithi002.vmouem (dependencies changed)... 2024-09-17T00:28:33.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:28:33 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi002.vmouem", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-17T00:28:33.731 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:28:33 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi002.vmouem", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-17T00:28:33.731 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:28:33 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:28:33.731 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:28:33 smithi002 ceph-mon[63804]: Reconfiguring daemon iscsi.foo.smithi002.vmouem on smithi002 2024-09-17T00:28:33.731 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:28:33 smithi002 ceph-mon[63804]: osdmap e142: 8 total, 8 up, 8 in 2024-09-17T00:28:34.279 INFO:teuthology.orchestra.run.smithi002.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-17T00:28:34.279 INFO:teuthology.orchestra.run.smithi002.stdout:alertmanager.a smithi002 *:9093,9094 running (6m) 81s ago 11m 26.1M - 0.25.0 c8568f914cd2 a21e13b3d677 2024-09-17T00:28:34.279 INFO:teuthology.orchestra.run.smithi002.stdout:grafana.a smithi099 *:3000 running (5m) 5s ago 11m 53.7M - 8.3.5 dad864ee21e9 bafb824a3c08 2024-09-17T00:28:34.279 INFO:teuthology.orchestra.run.smithi002.stdout:iscsi.foo.smithi002.vmouem smithi002 running (101s) 81s ago 10m 48.3M - 3.5 e1d6a67b021e b5424a0496a5 2024-09-17T00:28:34.279 INFO:teuthology.orchestra.run.smithi002.stdout:mgr.x smithi099 *:8443,9283,8765 running (5m) 5s ago 15m 442M - 19.1.1-330-gc2ddf972 ebb0615149de 6aa2ccef9ca0 2024-09-17T00:28:34.279 INFO:teuthology.orchestra.run.smithi002.stdout:mgr.y smithi002 *:8443,9283,8765 running (5m) 81s ago 17m 506M - 19.1.1-330-gc2ddf972 ebb0615149de 8838897e8449 2024-09-17T00:28:34.280 INFO:teuthology.orchestra.run.smithi002.stdout:mon.a smithi002 running (5m) 81s ago 17m 51.3M 2048M 19.1.1-330-gc2ddf972 ebb0615149de a035def53ef6 2024-09-17T00:28:34.280 INFO:teuthology.orchestra.run.smithi002.stdout:mon.b smithi099 running (4m) 5s ago 16m 41.2M 2048M 19.1.1-330-gc2ddf972 ebb0615149de aab3ab73a850 2024-09-17T00:28:34.280 INFO:teuthology.orchestra.run.smithi002.stdout:mon.c smithi002 running (4m) 81s ago 16m 43.4M 2048M 19.1.1-330-gc2ddf972 ebb0615149de 8a510255ae66 2024-09-17T00:28:34.280 INFO:teuthology.orchestra.run.smithi002.stdout:node-exporter.a smithi002 *:9100 running (6m) 81s ago 12m 14.8M - 1.5.0 0da6a335fe13 1030ae01bb32 2024-09-17T00:28:34.280 INFO:teuthology.orchestra.run.smithi002.stdout:node-exporter.b smithi099 *:9100 running (6m) 5s ago 12m 16.6M - 1.5.0 0da6a335fe13 01fcd35dca8a 2024-09-17T00:28:34.280 INFO:teuthology.orchestra.run.smithi002.stdout:osd.0 smithi002 running (3m) 81s ago 15m 45.8M 2737M 19.1.1-330-gc2ddf972 ebb0615149de 8345719cb6f0 2024-09-17T00:28:34.280 INFO:teuthology.orchestra.run.smithi002.stdout:osd.1 smithi002 running (2m) 81s ago 15m 51.5M 2737M 19.1.1-330-gc2ddf972 ebb0615149de 3dbc23d57c53 2024-09-17T00:28:34.280 INFO:teuthology.orchestra.run.smithi002.stdout:osd.2 smithi002 running (2m) 81s ago 14m 49.3M 2737M 19.1.1-330-gc2ddf972 ebb0615149de d21401b6763b 2024-09-17T00:28:34.280 INFO:teuthology.orchestra.run.smithi002.stdout:osd.3 smithi002 running (87s) 81s ago 14m 12.2M 2737M 19.1.1-330-gc2ddf972 ebb0615149de 980f2a25d096 2024-09-17T00:28:34.280 INFO:teuthology.orchestra.run.smithi002.stdout:osd.4 smithi099 running (48s) 5s ago 14m 47.8M 2992M 19.1.1-330-gc2ddf972 ebb0615149de f644e5be4db8 2024-09-17T00:28:34.280 INFO:teuthology.orchestra.run.smithi002.stdout:osd.5 smithi099 running (10s) 5s ago 13m 11.0M 2992M 19.1.1-330-gc2ddf972 ebb0615149de 2df0d88d16ce 2024-09-17T00:28:34.280 INFO:teuthology.orchestra.run.smithi002.stdout:osd.6 smithi099 running (13m) 5s ago 13m 58.7M 2992M 17.2.0 e1d6a67b021e 3213f5e6952a 2024-09-17T00:28:34.280 INFO:teuthology.orchestra.run.smithi002.stdout:osd.7 smithi099 running (12m) 5s ago 12m 65.0M 2992M 17.2.0 e1d6a67b021e 63be3a1ed0a1 2024-09-17T00:28:34.281 INFO:teuthology.orchestra.run.smithi002.stdout:prometheus.a smithi099 *:9095 running (5m) 5s ago 12m 67.2M - 2.43.0 a07b618ecd1d bf32c2ae11dc 2024-09-17T00:28:34.281 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.foo.smithi002.uwdhcs smithi002 *:8000 running (10m) 81s ago 10m 76.1M - 17.2.0 e1d6a67b021e 9c35813f459a 2024-09-17T00:28:34.281 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.foo.smithi099.utzfap smithi099 *:8000 running (10m) 5s ago 10m 76.9M - 17.2.0 e1d6a67b021e 34ec989160a6 2024-09-17T00:28:34.281 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.smpl.smithi002.rqnxii smithi002 *:80 running (10m) 81s ago 10m 74.4M - 17.2.0 e1d6a67b021e 0e180d0afb99 2024-09-17T00:28:34.281 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.smpl.smithi099.nzvkpm smithi099 *:80 running (10m) 5s ago 10m 77.5M - 17.2.0 e1d6a67b021e 6a068e82ac72 2024-09-17T00:28:34.697 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:28:34 smithi002 ceph-mon[66732]: from='client.34734 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:28:34.697 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:28:34 smithi002 ceph-mon[63804]: from='client.34734 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:28:34.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:28:34 smithi099 ceph-mon[56712]: from='client.34734 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:28:34.719 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-09-17T00:28:34.720 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": { 2024-09-17T00:28:34.720 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 19.1.1-330-gc2ddf972 (c2ddf9721ef254645201cdbabd9c8f49c30a586f) squid (stable)": 3 2024-09-17T00:28:34.720 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-09-17T00:28:34.720 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": { 2024-09-17T00:28:34.720 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 19.1.1-330-gc2ddf972 (c2ddf9721ef254645201cdbabd9c8f49c30a586f) squid (stable)": 2 2024-09-17T00:28:34.720 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-09-17T00:28:34.720 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": { 2024-09-17T00:28:34.720 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2, 2024-09-17T00:28:34.720 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 19.1.1-330-gc2ddf972 (c2ddf9721ef254645201cdbabd9c8f49c30a586f) squid (stable)": 6 2024-09-17T00:28:34.720 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-09-17T00:28:34.720 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw": { 2024-09-17T00:28:34.721 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-09-17T00:28:34.721 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-09-17T00:28:34.721 INFO:teuthology.orchestra.run.smithi002.stdout: "overall": { 2024-09-17T00:28:34.721 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 6, 2024-09-17T00:28:34.721 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 19.1.1-330-gc2ddf972 (c2ddf9721ef254645201cdbabd9c8f49c30a586f) squid (stable)": 11 2024-09-17T00:28:34.721 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-09-17T00:28:34.721 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-09-17T00:28:35.077 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-09-17T00:28:35.077 INFO:teuthology.orchestra.run.smithi002.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f", 2024-09-17T00:28:35.077 INFO:teuthology.orchestra.run.smithi002.stdout: "in_progress": true, 2024-09-17T00:28:35.077 INFO:teuthology.orchestra.run.smithi002.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-17T00:28:35.077 INFO:teuthology.orchestra.run.smithi002.stdout: "services_complete": [ 2024-09-17T00:28:35.078 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr", 2024-09-17T00:28:35.078 INFO:teuthology.orchestra.run.smithi002.stdout: "mon" 2024-09-17T00:28:35.078 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-09-17T00:28:35.078 INFO:teuthology.orchestra.run.smithi002.stdout: "progress": "11/23 daemons upgraded", 2024-09-17T00:28:35.078 INFO:teuthology.orchestra.run.smithi002.stdout: "message": "Currently upgrading osd daemons", 2024-09-17T00:28:35.078 INFO:teuthology.orchestra.run.smithi002.stdout: "is_paused": false 2024-09-17T00:28:35.078 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-09-17T00:28:35.499 INFO:teuthology.orchestra.run.smithi002.stdout:HEALTH_WARN Degraded data redundancy: 60/726 objects degraded (8.264%), 14 pgs degraded 2024-09-17T00:28:35.499 INFO:teuthology.orchestra.run.smithi002.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 60/726 objects degraded (8.264%), 14 pgs degraded 2024-09-17T00:28:35.499 INFO:teuthology.orchestra.run.smithi002.stdout: pg 2.2 is active+undersized+degraded, acting [1,6] 2024-09-17T00:28:35.500 INFO:teuthology.orchestra.run.smithi002.stdout: pg 2.12 is active+undersized+degraded, acting [3,2] 2024-09-17T00:28:35.500 INFO:teuthology.orchestra.run.smithi002.stdout: pg 2.15 is active+undersized+degraded, acting [1,0] 2024-09-17T00:28:35.500 INFO:teuthology.orchestra.run.smithi002.stdout: pg 2.16 is active+undersized+degraded, acting [6,2] 2024-09-17T00:28:35.500 INFO:teuthology.orchestra.run.smithi002.stdout: pg 3.4 is active+undersized+degraded, acting [1,2] 2024-09-17T00:28:35.500 INFO:teuthology.orchestra.run.smithi002.stdout: pg 3.5 is active+undersized+degraded, acting [3,2] 2024-09-17T00:28:35.500 INFO:teuthology.orchestra.run.smithi002.stdout: pg 3.c is active+undersized+degraded, acting [3,6] 2024-09-17T00:28:35.500 INFO:teuthology.orchestra.run.smithi002.stdout: pg 3.16 is active+undersized+degraded, acting [7,1] 2024-09-17T00:28:35.500 INFO:teuthology.orchestra.run.smithi002.stdout: pg 3.17 is active+undersized+degraded, acting [0,3] 2024-09-17T00:28:35.500 INFO:teuthology.orchestra.run.smithi002.stdout: pg 3.1c is active+undersized+degraded, acting [4,1] 2024-09-17T00:28:35.500 INFO:teuthology.orchestra.run.smithi002.stdout: pg 3.1d is active+undersized+degraded, acting [4,6] 2024-09-17T00:28:35.500 INFO:teuthology.orchestra.run.smithi002.stdout: pg 3.1f is active+undersized+degraded, acting [0,2] 2024-09-17T00:28:35.500 INFO:teuthology.orchestra.run.smithi002.stdout: pg 4.3 is active+undersized+degraded, acting [0,7] 2024-09-17T00:28:35.501 INFO:teuthology.orchestra.run.smithi002.stdout: pg 4.15 is active+undersized+degraded, acting [7,3] 2024-09-17T00:28:35.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:28:35 smithi099 ceph-mon[56712]: from='client.44647 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:28:35.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:28:35 smithi099 ceph-mon[56712]: pgmap v188: 161 pgs: 15 peering, 25 active+undersized, 14 active+undersized+degraded, 107 active+clean; 585 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 60/726 objects degraded (8.264%) 2024-09-17T00:28:35.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:28:35 smithi099 ceph-mon[56712]: from='client.44653 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:28:35.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:28:35 smithi099 ceph-mon[56712]: Health check update: Degraded data redundancy: 60/726 objects degraded (8.264%), 14 pgs degraded (PG_DEGRADED) 2024-09-17T00:28:35.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:28:35 smithi099 ceph-mon[56712]: from='client.? 172.21.15.2:0/3925924993' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:28:35.725 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:28:35 smithi002 ceph-mon[66732]: from='client.44647 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:28:35.725 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:28:35 smithi002 ceph-mon[66732]: pgmap v188: 161 pgs: 15 peering, 25 active+undersized, 14 active+undersized+degraded, 107 active+clean; 585 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 60/726 objects degraded (8.264%) 2024-09-17T00:28:35.725 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:28:35 smithi002 ceph-mon[66732]: from='client.44653 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:28:35.725 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:28:35 smithi002 ceph-mon[66732]: Health check update: Degraded data redundancy: 60/726 objects degraded (8.264%), 14 pgs degraded (PG_DEGRADED) 2024-09-17T00:28:35.725 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:28:35 smithi002 ceph-mon[66732]: from='client.? 172.21.15.2:0/3925924993' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:28:35.725 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:28:35 smithi002 ceph-mon[63804]: from='client.44647 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:28:35.725 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:28:35 smithi002 ceph-mon[63804]: pgmap v188: 161 pgs: 15 peering, 25 active+undersized, 14 active+undersized+degraded, 107 active+clean; 585 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 60/726 objects degraded (8.264%) 2024-09-17T00:28:35.725 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:28:35 smithi002 ceph-mon[63804]: from='client.44653 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:28:35.726 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:28:35 smithi002 ceph-mon[63804]: Health check update: Degraded data redundancy: 60/726 objects degraded (8.264%), 14 pgs degraded (PG_DEGRADED) 2024-09-17T00:28:35.726 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:28:35 smithi002 ceph-mon[63804]: from='client.? 172.21.15.2:0/3925924993' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:28:36.641 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:28:36 smithi002 ceph-mon[66732]: from='client.44665 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:28:36.641 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:28:36 smithi002 ceph-mon[66732]: from='client.? 172.21.15.2:0/1077584934' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T00:28:36.641 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:28:36 smithi002 ceph-mon[63804]: from='client.44665 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:28:36.641 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:28:36 smithi002 ceph-mon[63804]: from='client.? 172.21.15.2:0/1077584934' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T00:28:36.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:28:36 smithi099 ceph-mon[56712]: from='client.44665 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:28:36.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:28:36 smithi099 ceph-mon[56712]: from='client.? 172.21.15.2:0/1077584934' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T00:28:36.748 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:28:36 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: ::ffff:172.21.15.99 - - [17/Sep/2024:00:28:36] "GET /metrics HTTP/1.1" 200 38459 "" "Prometheus/2.43.0" 2024-09-17T00:28:37.272 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:28:37 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: 2024-09-17T00:28:37.196+0000 7f17efec7640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (9 PGs are or would become offline) 2024-09-17T00:28:37.580 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:28:37 smithi002 ceph-mon[63804]: pgmap v189: 161 pgs: 15 peering, 8 active+undersized, 5 active+undersized+degraded, 133 active+clean; 585 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 23/726 objects degraded (3.168%) 2024-09-17T00:28:37.580 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:28:37 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:28:37.580 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:28:37 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:28:37.580 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:28:37 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-09-17T00:28:37.580 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:28:37 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-09-17T00:28:37.580 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:28:37 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T00:28:37.580 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:28:37 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:28:37.580 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:28:37 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:28:37.580 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:28:37 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:28:37.580 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:28:37 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-09-17T00:28:37.581 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:28:37 smithi002 ceph-mon[66732]: pgmap v189: 161 pgs: 15 peering, 8 active+undersized, 5 active+undersized+degraded, 133 active+clean; 585 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 23/726 objects degraded (3.168%) 2024-09-17T00:28:37.581 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:28:37 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:28:37.581 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:28:37 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:28:37.581 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:28:37 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-09-17T00:28:37.581 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:28:37 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-09-17T00:28:37.581 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:28:37 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T00:28:37.581 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:28:37 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:28:37.581 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:28:37 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:28:37.581 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:28:37 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:28:37.581 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:28:37 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-09-17T00:28:37.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:28:37 smithi099 ceph-mon[56712]: pgmap v189: 161 pgs: 15 peering, 8 active+undersized, 5 active+undersized+degraded, 133 active+clean; 585 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 23/726 objects degraded (3.168%) 2024-09-17T00:28:37.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:28:37 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:28:37.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:28:37 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:28:37.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:28:37 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-09-17T00:28:37.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:28:37 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-09-17T00:28:37.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:28:37 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T00:28:37.718 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:28:37 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:28:37.718 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:28:37 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:28:37.718 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:28:37 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:28:37.718 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:28:37 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-09-17T00:28:38.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:28:38 smithi002 ceph-mon[63804]: from='mon.1 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-09-17T00:28:38.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:28:38 smithi002 ceph-mon[63804]: from='mon.1 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-09-17T00:28:38.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:28:38 smithi002 ceph-mon[63804]: from='mon.1 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-09-17T00:28:38.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:28:38 smithi002 ceph-mon[63804]: Upgrade: unsafe to stop osd(s) at this time (9 PGs are or would become offline) 2024-09-17T00:28:38.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:28:38 smithi002 ceph-mon[63804]: from='client.? 172.21.15.2:0/474644542' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-17T00:28:38.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:28:38 smithi002 ceph-mon[66732]: from='mon.1 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-09-17T00:28:38.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:28:38 smithi002 ceph-mon[66732]: from='mon.1 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-09-17T00:28:38.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:28:38 smithi002 ceph-mon[66732]: from='mon.1 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-09-17T00:28:38.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:28:38 smithi002 ceph-mon[66732]: Upgrade: unsafe to stop osd(s) at this time (9 PGs are or would become offline) 2024-09-17T00:28:38.731 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:28:38 smithi002 ceph-mon[66732]: from='client.? 172.21.15.2:0/474644542' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-17T00:28:38.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:28:38 smithi099 ceph-mon[56712]: from='mon.1 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-09-17T00:28:38.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:28:38 smithi099 ceph-mon[56712]: from='mon.1 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-09-17T00:28:38.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:28:38 smithi099 ceph-mon[56712]: from='mon.1 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-09-17T00:28:38.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:28:38 smithi099 ceph-mon[56712]: Upgrade: unsafe to stop osd(s) at this time (9 PGs are or would become offline) 2024-09-17T00:28:38.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:28:38 smithi099 ceph-mon[56712]: from='client.? 172.21.15.2:0/474644542' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-17T00:28:39.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:28:39 smithi002 ceph-mon[63804]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 787 B/s rd, 0 op/s 2024-09-17T00:28:39.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:28:39 smithi002 ceph-mon[63804]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 23/726 objects degraded (3.168%), 5 pgs degraded) 2024-09-17T00:28:39.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:28:39 smithi002 ceph-mon[63804]: Cluster is now healthy 2024-09-17T00:28:39.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:28:39 smithi002 ceph-mon[66732]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 787 B/s rd, 0 op/s 2024-09-17T00:28:39.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:28:39 smithi002 ceph-mon[66732]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 23/726 objects degraded (3.168%), 5 pgs degraded) 2024-09-17T00:28:39.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:28:39 smithi002 ceph-mon[66732]: Cluster is now healthy 2024-09-17T00:28:39.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:28:39 smithi099 ceph-mon[56712]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 787 B/s rd, 0 op/s 2024-09-17T00:28:39.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:28:39 smithi099 ceph-mon[56712]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 23/726 objects degraded (3.168%), 5 pgs degraded) 2024-09-17T00:28:39.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:28:39 smithi099 ceph-mon[56712]: Cluster is now healthy 2024-09-17T00:28:41.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:28:41 smithi002 ceph-mon[63804]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:28:41.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:28:41 smithi002 ceph-mon[66732]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:28:41.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:28:41 smithi099 ceph-mon[56712]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:28:43.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:28:42 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:28:43.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:28:42 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T00:28:43.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:28:42 smithi099 ceph-mon[56712]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2024-09-17T00:28:43.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:28:42 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:28:43.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:28:42 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T00:28:43.230 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:28:42 smithi002 ceph-mon[66732]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2024-09-17T00:28:43.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:28:42 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:28:43.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:28:42 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T00:28:43.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:28:42 smithi002 ceph-mon[63804]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2024-09-17T00:28:45.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:28:45 smithi099 ceph-mon[56712]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-17T00:28:45.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:28:45 smithi002 ceph-mon[63804]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-17T00:28:45.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:28:45 smithi002 ceph-mon[66732]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-17T00:28:46.979 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:28:46 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: ::ffff:172.21.15.99 - - [17/Sep/2024:00:28:46] "GET /metrics HTTP/1.1" 200 38467 "" "Prometheus/2.43.0" 2024-09-17T00:28:47.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:28:47 smithi099 ceph-mon[56712]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-17T00:28:47.783 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:28:47 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:28:47.784 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:28:47 smithi002 ceph-mon[63804]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-17T00:28:47.784 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:28:47 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:28:47.784 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:28:47 smithi002 ceph-mon[66732]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-17T00:28:47.784 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:28:47 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:28:48.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:28:48 smithi099 ceph-mon[56712]: from='client.54506 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:28:48.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:28:48 smithi002 ceph-mon[66732]: from='client.54506 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:28:48.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:28:48 smithi002 ceph-mon[63804]: from='client.54506 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:28:49.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:28:49 smithi099 ceph-mon[56712]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-17T00:28:49.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:28:49 smithi002 ceph-mon[66732]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-17T00:28:49.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:28:49 smithi002 ceph-mon[63804]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-17T00:28:51.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:28:51 smithi099 ceph-mon[56712]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:28:51.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:28:51 smithi002 ceph-mon[66732]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:28:51.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:28:51 smithi002 ceph-mon[63804]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:28:52.821 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:28:52 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-09-17T00:28:52.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:28:52 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-09-17T00:28:52.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:28:52 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-09-17T00:28:53.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:28:53 smithi099 ceph-mon[56712]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:28:53.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:28:53 smithi099 ceph-mon[56712]: from='mon.1 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-09-17T00:28:53.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:28:53 smithi099 ceph-mon[56712]: Upgrade: osd.6 is safe to restart 2024-09-17T00:28:53.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:28:53 smithi002 ceph-mon[66732]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:28:53.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:28:53 smithi002 ceph-mon[66732]: from='mon.1 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-09-17T00:28:53.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:28:53 smithi002 ceph-mon[66732]: Upgrade: osd.6 is safe to restart 2024-09-17T00:28:53.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:28:53 smithi002 ceph-mon[63804]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:28:53.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:28:53 smithi002 ceph-mon[63804]: from='mon.1 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-09-17T00:28:53.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:28:53 smithi002 ceph-mon[63804]: Upgrade: osd.6 is safe to restart 2024-09-17T00:28:54.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:28:54 smithi099 ceph-mon[56712]: Upgrade: Updating osd.6 2024-09-17T00:28:54.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:28:54 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:28:54.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:28:54 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-09-17T00:28:54.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:28:54 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:28:54.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:28:54 smithi099 ceph-mon[56712]: Deploying daemon osd.6 on smithi099 2024-09-17T00:28:54.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:28:54 smithi099 ceph-mon[56712]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:28:54.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:28:54 smithi002 ceph-mon[66732]: Upgrade: Updating osd.6 2024-09-17T00:28:54.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:28:54 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:28:54.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:28:54 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-09-17T00:28:54.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:28:54 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:28:54.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:28:54 smithi002 ceph-mon[66732]: Deploying daemon osd.6 on smithi099 2024-09-17T00:28:54.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:28:54 smithi002 ceph-mon[66732]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:28:54.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:28:54 smithi002 ceph-mon[63804]: Upgrade: Updating osd.6 2024-09-17T00:28:54.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:28:54 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:28:54.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:28:54 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-09-17T00:28:54.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:28:54 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:28:54.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:28:54 smithi002 ceph-mon[63804]: Deploying daemon osd.6 on smithi099 2024-09-17T00:28:54.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:28:54 smithi002 ceph-mon[63804]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:28:56.760 INFO:journalctl@ceph.osd.6.smithi099.stdout:Sep 17 00:28:56 smithi099 systemd[1]: Stopping Ceph osd.6 for 3aad27a2-7489-11ef-bceb-c7b262605968... 2024-09-17T00:28:56.979 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:28:56 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: ::ffff:172.21.15.99 - - [17/Sep/2024:00:28:56] "GET /metrics HTTP/1.1" 200 38467 "" "Prometheus/2.43.0" 2024-09-17T00:28:57.217 INFO:journalctl@ceph.osd.6.smithi099.stdout:Sep 17 00:28:56 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-6[33723]: 2024-09-17T00:28:56.762+0000 7ff82fade700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-17T00:28:57.217 INFO:journalctl@ceph.osd.6.smithi099.stdout:Sep 17 00:28:56 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-6[33723]: 2024-09-17T00:28:56.762+0000 7ff82fade700 -1 osd.6 142 *** Got signal Terminated *** 2024-09-17T00:28:57.217 INFO:journalctl@ceph.osd.6.smithi099.stdout:Sep 17 00:28:56 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-6[33723]: 2024-09-17T00:28:56.762+0000 7ff82fade700 -1 osd.6 142 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-17T00:28:57.671 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:28:57 smithi099 ceph-mon[56712]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:28:57.671 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:28:57 smithi099 ceph-mon[56712]: osd.6 marked itself down and dead 2024-09-17T00:28:57.671 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:28:57 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T00:28:57.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:28:57 smithi002 ceph-mon[63804]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:28:57.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:28:57 smithi002 ceph-mon[63804]: osd.6 marked itself down and dead 2024-09-17T00:28:57.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:28:57 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T00:28:57.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:28:57 smithi002 ceph-mon[66732]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:28:57.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:28:57 smithi002 ceph-mon[66732]: osd.6 marked itself down and dead 2024-09-17T00:28:57.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:28:57 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T00:28:57.967 INFO:journalctl@ceph.osd.6.smithi099.stdout:Sep 17 00:28:57 smithi099 podman[64121]: 2024-09-17 00:28:57.708107805 +0000 UTC m=+1.050405685 container died 3213f5e6952a1eee2f62d4a98de06891bcd2ba2777f297637d1d01b31f11cdd5 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-6, io.openshift.expose-services=, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, maintainer=Guillaume Abrioux , description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_CLEAN=True, version=8, architecture=x86_64, distribution-scope=public, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_BRANCH=HEAD, io.k8s.display-name=CentOS Stream 8, io.openshift.tags=base centos centos-stream, vcs-type=git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, ceph=True, release=754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, build-date=2022-05-03T08:36:31.336870, vendor=Red Hat, Inc., name=centos-stream, io.buildah.version=1.19.8, RELEASE=HEAD, com.redhat.component=centos-stream-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-17.2.0) 2024-09-17T00:28:58.277 INFO:journalctl@ceph.osd.6.smithi099.stdout:Sep 17 00:28:58 smithi099 podman[64121]: 2024-09-17 00:28:58.079501944 +0000 UTC m=+1.421799824 container cleanup 3213f5e6952a1eee2f62d4a98de06891bcd2ba2777f297637d1d01b31f11cdd5 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-6, io.k8s.display-name=CentOS Stream 8, com.redhat.component=centos-stream-container, distribution-scope=public, vendor=Red Hat, Inc., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, name=centos-stream, CEPH_POINT_RELEASE=-17.2.0, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, maintainer=Guillaume Abrioux , io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-05-03T08:36:31.336870, ceph=True, RELEASE=HEAD, io.openshift.tags=base centos centos-stream, vcs-type=git, release=754, architecture=x86_64, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.buildah.version=1.19.8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_BRANCH=HEAD, version=8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_CLEAN=True, io.openshift.expose-services=, GIT_REPO=https://github.com/ceph/ceph-container.git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754) 2024-09-17T00:28:58.277 INFO:journalctl@ceph.osd.6.smithi099.stdout:Sep 17 00:28:58 smithi099 bash[64121]: ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-6 2024-09-17T00:28:58.695 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:28:58 smithi099 ceph-mon[56712]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-17T00:28:58.696 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:28:58 smithi099 ceph-mon[56712]: osdmap e143: 8 total, 7 up, 8 in 2024-09-17T00:28:58.696 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:28:58 smithi099 ceph-mon[56712]: from='client.54506 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:28:58.696 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:28:58 smithi099 ceph-mon[56712]: osdmap e144: 8 total, 7 up, 8 in 2024-09-17T00:28:58.696 INFO:journalctl@ceph.osd.6.smithi099.stdout:Sep 17 00:28:58 smithi099 podman[64146]: 2024-09-17 00:28:58.381279376 +0000 UTC m=+0.674109692 container remove 3213f5e6952a1eee2f62d4a98de06891bcd2ba2777f297637d1d01b31f11cdd5 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-6, name=centos-stream, distribution-scope=public, vcs-type=git, io.openshift.tags=base centos centos-stream, ceph=True, io.buildah.version=1.19.8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., version=8, build-date=2022-05-03T08:36:31.336870, maintainer=Guillaume Abrioux , GIT_CLEAN=True, release=754, architecture=x86_64, com.redhat.component=centos-stream-container, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, CEPH_POINT_RELEASE=-17.2.0, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.display-name=CentOS Stream 8, io.openshift.expose-services=, vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, RELEASE=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.license_terms=https://centos.org/legal/licensing-policy/) 2024-09-17T00:28:58.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:28:58 smithi002 ceph-mon[63804]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-17T00:28:58.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:28:58 smithi002 ceph-mon[63804]: osdmap e143: 8 total, 7 up, 8 in 2024-09-17T00:28:58.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:28:58 smithi002 ceph-mon[63804]: from='client.54506 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:28:58.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:28:58 smithi002 ceph-mon[63804]: osdmap e144: 8 total, 7 up, 8 in 2024-09-17T00:28:58.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:28:58 smithi002 ceph-mon[66732]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-17T00:28:58.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:28:58 smithi002 ceph-mon[66732]: osdmap e143: 8 total, 7 up, 8 in 2024-09-17T00:28:58.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:28:58 smithi002 ceph-mon[66732]: from='client.54506 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:28:58.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:28:58 smithi002 ceph-mon[66732]: osdmap e144: 8 total, 7 up, 8 in 2024-09-17T00:28:58.965 INFO:journalctl@ceph.osd.6.smithi099.stdout:Sep 17 00:28:58 smithi099 podman[64236]: 2024-09-17 00:28:58.698879611 +0000 UTC m=+0.113509230 container create 2dca4b159a285cd721a76af7a578831b0b52621071a0e7e960f4d364da2bc74c (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-6-deactivate, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=squid-c2ddf97, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, ceph=True) 2024-09-17T00:28:58.965 INFO:journalctl@ceph.osd.6.smithi099.stdout:Sep 17 00:28:58 smithi099 podman[64236]: 2024-09-17 00:28:58.60206821 +0000 UTC m=+0.016697846 image pull ebb0615149def4a3179c82c0cc47af75808300abb9f1142788e5e605a7ac0e5b quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f 2024-09-17T00:28:58.965 INFO:journalctl@ceph.osd.6.smithi099.stdout:Sep 17 00:28:58 smithi099 podman[64236]: 2024-09-17 00:28:58.893102263 +0000 UTC m=+0.307731901 container init 2dca4b159a285cd721a76af7a578831b0b52621071a0e7e960f4d364da2bc74c (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-6-deactivate, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, RELEASE=squid-c2ddf97, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-09-17T00:28:58.965 INFO:journalctl@ceph.osd.6.smithi099.stdout:Sep 17 00:28:58 smithi099 podman[64236]: 2024-09-17 00:28:58.896417429 +0000 UTC m=+0.311047048 container start 2dca4b159a285cd721a76af7a578831b0b52621071a0e7e960f4d364da2bc74c (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-6-deactivate, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-c2ddf97, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-17T00:28:58.965 INFO:journalctl@ceph.osd.6.smithi099.stdout:Sep 17 00:28:58 smithi099 podman[64236]: 2024-09-17 00:28:58.968153117 +0000 UTC m=+0.382782751 container attach 2dca4b159a285cd721a76af7a578831b0b52621071a0e7e960f4d364da2bc74c (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-6-deactivate, org.label-schema.license=GPLv2, RELEASE=squid-c2ddf97, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=) 2024-09-17T00:28:59.217 INFO:journalctl@ceph.osd.6.smithi099.stdout:Sep 17 00:28:59 smithi099 podman[64236]: 2024-09-17 00:28:59.081389767 +0000 UTC m=+0.496019393 container died 2dca4b159a285cd721a76af7a578831b0b52621071a0e7e960f4d364da2bc74c (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-6-deactivate, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=squid-c2ddf97, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-17T00:28:59.694 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:28:59 smithi099 ceph-mon[56712]: pgmap v201: 161 pgs: 25 peering, 6 stale+active+clean, 130 active+clean; 585 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-09-17T00:28:59.695 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:28:59 smithi099 ceph-mon[56712]: Health check failed: Reduced data availability: 6 pgs inactive, 6 pgs peering (PG_AVAILABILITY) 2024-09-17T00:28:59.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:28:59 smithi002 ceph-mon[66732]: pgmap v201: 161 pgs: 25 peering, 6 stale+active+clean, 130 active+clean; 585 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-09-17T00:28:59.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:28:59 smithi002 ceph-mon[66732]: Health check failed: Reduced data availability: 6 pgs inactive, 6 pgs peering (PG_AVAILABILITY) 2024-09-17T00:28:59.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:28:59 smithi002 ceph-mon[63804]: pgmap v201: 161 pgs: 25 peering, 6 stale+active+clean, 130 active+clean; 585 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-09-17T00:28:59.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:28:59 smithi002 ceph-mon[63804]: Health check failed: Reduced data availability: 6 pgs inactive, 6 pgs peering (PG_AVAILABILITY) 2024-09-17T00:28:59.967 INFO:journalctl@ceph.osd.6.smithi099.stdout:Sep 17 00:28:59 smithi099 podman[64259]: 2024-09-17 00:28:59.697446248 +0000 UTC m=+0.609474743 container remove 2dca4b159a285cd721a76af7a578831b0b52621071a0e7e960f4d364da2bc74c (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-6-deactivate, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, RELEASE=squid-c2ddf97, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-09-17T00:28:59.967 INFO:journalctl@ceph.osd.6.smithi099.stdout:Sep 17 00:28:59 smithi099 systemd[1]: ceph-3aad27a2-7489-11ef-bceb-c7b262605968@osd.6.service: Deactivated successfully. 2024-09-17T00:28:59.967 INFO:journalctl@ceph.osd.6.smithi099.stdout:Sep 17 00:28:59 smithi099 systemd[1]: Stopped Ceph osd.6 for 3aad27a2-7489-11ef-bceb-c7b262605968. 2024-09-17T00:28:59.967 INFO:journalctl@ceph.osd.6.smithi099.stdout:Sep 17 00:28:59 smithi099 systemd[1]: ceph-3aad27a2-7489-11ef-bceb-c7b262605968@osd.6.service: Consumed 8.978s CPU time. 2024-09-17T00:28:59.968 INFO:journalctl@ceph.osd.6.smithi099.stdout:Sep 17 00:28:59 smithi099 systemd[1]: Starting Ceph osd.6 for 3aad27a2-7489-11ef-bceb-c7b262605968... 2024-09-17T00:29:00.423 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:00 smithi099 ceph-mon[56712]: Health check failed: Degraded data redundancy: 14/726 objects degraded (1.928%), 4 pgs degraded (PG_DEGRADED) 2024-09-17T00:29:00.424 INFO:journalctl@ceph.osd.6.smithi099.stdout:Sep 17 00:29:00 smithi099 podman[64358]: 2024-09-17 00:29:00.029297799 +0000 UTC m=+0.015916398 image pull ebb0615149def4a3179c82c0cc47af75808300abb9f1142788e5e605a7ac0e5b quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f 2024-09-17T00:29:00.424 INFO:journalctl@ceph.osd.6.smithi099.stdout:Sep 17 00:29:00 smithi099 podman[64358]: 2024-09-17 00:29:00.157131771 +0000 UTC m=+0.143750359 container create 16c11d5b966148d1702589380196bfa77b281881481bdc7fb769103e665b4db6 (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-6-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-c2ddf97, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-09-17T00:29:00.424 INFO:journalctl@ceph.osd.6.smithi099.stdout:Sep 17 00:29:00 smithi099 podman[64358]: 2024-09-17 00:29:00.426646897 +0000 UTC m=+0.413265496 container init 16c11d5b966148d1702589380196bfa77b281881481bdc7fb769103e665b4db6 (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-6-activate, RELEASE=squid-c2ddf97, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-09-17T00:29:00.717 INFO:journalctl@ceph.osd.6.smithi099.stdout:Sep 17 00:29:00 smithi099 podman[64358]: 2024-09-17 00:29:00.430000254 +0000 UTC m=+0.416618844 container start 16c11d5b966148d1702589380196bfa77b281881481bdc7fb769103e665b4db6 (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-6-activate, io.buildah.version=1.37.2, RELEASE=squid-c2ddf97, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-09-17T00:29:00.717 INFO:journalctl@ceph.osd.6.smithi099.stdout:Sep 17 00:29:00 smithi099 podman[64358]: 2024-09-17 00:29:00.518385562 +0000 UTC m=+0.505004154 container attach 16c11d5b966148d1702589380196bfa77b281881481bdc7fb769103e665b4db6 (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-6-activate, RELEASE=squid-c2ddf97, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, GIT_CLEAN=True, io.buildah.version=1.37.2) 2024-09-17T00:29:00.717 INFO:journalctl@ceph.osd.6.smithi099.stdout:Sep 17 00:29:00 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-6-activate[64374]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-17T00:29:00.717 INFO:journalctl@ceph.osd.6.smithi099.stdout:Sep 17 00:29:00 smithi099 bash[64358]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-17T00:29:00.717 INFO:journalctl@ceph.osd.6.smithi099.stdout:Sep 17 00:29:00 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-6-activate[64374]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-17T00:29:00.717 INFO:journalctl@ceph.osd.6.smithi099.stdout:Sep 17 00:29:00 smithi099 bash[64358]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-17T00:29:00.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:00 smithi002 ceph-mon[63804]: Health check failed: Degraded data redundancy: 14/726 objects degraded (1.928%), 4 pgs degraded (PG_DEGRADED) 2024-09-17T00:29:00.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:00 smithi002 ceph-mon[66732]: Health check failed: Degraded data redundancy: 14/726 objects degraded (1.928%), 4 pgs degraded (PG_DEGRADED) 2024-09-17T00:29:01.619 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:01 smithi099 ceph-mon[56712]: pgmap v203: 161 pgs: 7 active+undersized, 25 peering, 5 stale+active+clean, 4 active+undersized+degraded, 120 active+clean; 585 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 14/726 objects degraded (1.928%) 2024-09-17T00:29:01.619 INFO:journalctl@ceph.osd.6.smithi099.stdout:Sep 17 00:29:01 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-6-activate[64374]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-09-17T00:29:01.619 INFO:journalctl@ceph.osd.6.smithi099.stdout:Sep 17 00:29:01 smithi099 bash[64358]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-09-17T00:29:01.619 INFO:journalctl@ceph.osd.6.smithi099.stdout:Sep 17 00:29:01 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-6-activate[64374]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-6 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-09-17T00:29:01.619 INFO:journalctl@ceph.osd.6.smithi099.stdout:Sep 17 00:29:01 smithi099 bash[64358]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-6 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-09-17T00:29:01.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:01 smithi002 ceph-mon[66732]: pgmap v203: 161 pgs: 7 active+undersized, 25 peering, 5 stale+active+clean, 4 active+undersized+degraded, 120 active+clean; 585 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 14/726 objects degraded (1.928%) 2024-09-17T00:29:01.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:01 smithi002 ceph-mon[63804]: pgmap v203: 161 pgs: 7 active+undersized, 25 peering, 5 stale+active+clean, 4 active+undersized+degraded, 120 active+clean; 585 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 14/726 objects degraded (1.928%) 2024-09-17T00:29:01.967 INFO:journalctl@ceph.osd.6.smithi099.stdout:Sep 17 00:29:01 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-6-activate[64374]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-09-17T00:29:01.967 INFO:journalctl@ceph.osd.6.smithi099.stdout:Sep 17 00:29:01 smithi099 bash[64358]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-09-17T00:29:01.967 INFO:journalctl@ceph.osd.6.smithi099.stdout:Sep 17 00:29:01 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-6-activate[64374]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-09-17T00:29:01.967 INFO:journalctl@ceph.osd.6.smithi099.stdout:Sep 17 00:29:01 smithi099 bash[64358]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-09-17T00:29:01.967 INFO:journalctl@ceph.osd.6.smithi099.stdout:Sep 17 00:29:01 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-6-activate[64374]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-6/block 2024-09-17T00:29:01.968 INFO:journalctl@ceph.osd.6.smithi099.stdout:Sep 17 00:29:01 smithi099 bash[64358]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-6/block 2024-09-17T00:29:01.968 INFO:journalctl@ceph.osd.6.smithi099.stdout:Sep 17 00:29:01 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-6-activate[64374]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-09-17T00:29:01.968 INFO:journalctl@ceph.osd.6.smithi099.stdout:Sep 17 00:29:01 smithi099 bash[64358]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-09-17T00:29:01.968 INFO:journalctl@ceph.osd.6.smithi099.stdout:Sep 17 00:29:01 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-6-activate[64374]: --> ceph-volume raw activate successful for osd ID: 6 2024-09-17T00:29:01.968 INFO:journalctl@ceph.osd.6.smithi099.stdout:Sep 17 00:29:01 smithi099 bash[64358]: --> ceph-volume raw activate successful for osd ID: 6 2024-09-17T00:29:01.968 INFO:journalctl@ceph.osd.6.smithi099.stdout:Sep 17 00:29:01 smithi099 podman[64358]: 2024-09-17 00:29:01.65061419 +0000 UTC m=+1.637232794 container died 16c11d5b966148d1702589380196bfa77b281881481bdc7fb769103e665b4db6 (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-6-activate, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=squid-c2ddf97, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-17T00:29:02.417 INFO:journalctl@ceph.osd.6.smithi099.stdout:Sep 17 00:29:02 smithi099 podman[64517]: 2024-09-17 00:29:02.213934364 +0000 UTC m=+0.557467884 container remove 16c11d5b966148d1702589380196bfa77b281881481bdc7fb769103e665b4db6 (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-6-activate, ceph=True, RELEASE=squid-c2ddf97, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-17T00:29:02.680 INFO:journalctl@ceph.osd.6.smithi099.stdout:Sep 17 00:29:02 smithi099 podman[64577]: 2024-09-17 00:29:02.322784687 +0000 UTC m=+0.015820476 image pull ebb0615149def4a3179c82c0cc47af75808300abb9f1142788e5e605a7ac0e5b quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f 2024-09-17T00:29:02.681 INFO:journalctl@ceph.osd.6.smithi099.stdout:Sep 17 00:29:02 smithi099 podman[64577]: 2024-09-17 00:29:02.440061584 +0000 UTC m=+0.133097407 container create 7462bdbdf29b7ec691c9bfa3dc9777f167fc9ab0c3a8e6663e289c2a3ca8f1bd (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-6, ceph=True, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, RELEASE=squid-c2ddf97, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-09-17T00:29:02.681 INFO:journalctl@ceph.osd.6.smithi099.stdout:Sep 17 00:29:02 smithi099 podman[64577]: 2024-09-17 00:29:02.658848359 +0000 UTC m=+0.351884137 container init 7462bdbdf29b7ec691c9bfa3dc9777f167fc9ab0c3a8e6663e289c2a3ca8f1bd (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-6, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=squid-c2ddf97) 2024-09-17T00:29:02.681 INFO:journalctl@ceph.osd.6.smithi099.stdout:Sep 17 00:29:02 smithi099 podman[64577]: 2024-09-17 00:29:02.661960829 +0000 UTC m=+0.354996607 container start 7462bdbdf29b7ec691c9bfa3dc9777f167fc9ab0c3a8e6663e289c2a3ca8f1bd (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-6, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , RELEASE=squid-c2ddf97, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-09-17T00:29:02.967 INFO:journalctl@ceph.osd.6.smithi099.stdout:Sep 17 00:29:02 smithi099 ceph-osd[64596]: -- 172.21.15.99:0/2267425550 <== mon.2 v2:172.21.15.99:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56124a495a40 con 0x56124b25c000 2024-09-17T00:29:02.967 INFO:journalctl@ceph.osd.6.smithi099.stdout:Sep 17 00:29:02 smithi099 bash[64577]: 7462bdbdf29b7ec691c9bfa3dc9777f167fc9ab0c3a8e6663e289c2a3ca8f1bd 2024-09-17T00:29:02.967 INFO:journalctl@ceph.osd.6.smithi099.stdout:Sep 17 00:29:02 smithi099 systemd[1]: Started Ceph osd.6 for 3aad27a2-7489-11ef-bceb-c7b262605968. 2024-09-17T00:29:03.597 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:03 smithi002 ceph-mon[66732]: pgmap v204: 161 pgs: 17 active+undersized, 25 peering, 11 active+undersized+degraded, 108 active+clean; 585 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s; 42/726 objects degraded (5.785%) 2024-09-17T00:29:03.597 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:03 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:29:03.597 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:03 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:29:03.598 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:03 smithi002 ceph-mon[63804]: pgmap v204: 161 pgs: 17 active+undersized, 25 peering, 11 active+undersized+degraded, 108 active+clean; 585 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s; 42/726 objects degraded (5.785%) 2024-09-17T00:29:03.598 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:03 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:29:03.598 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:03 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:29:03.642 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:03 smithi099 ceph-mon[56712]: pgmap v204: 161 pgs: 17 active+undersized, 25 peering, 11 active+undersized+degraded, 108 active+clean; 585 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s; 42/726 objects degraded (5.785%) 2024-09-17T00:29:03.642 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:03 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:29:03.643 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:03 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:29:04.685 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:04 smithi002 ceph-mon[66732]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 6 pgs inactive, 6 pgs peering) 2024-09-17T00:29:04.686 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:04 smithi002 ceph-mon[63804]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 6 pgs inactive, 6 pgs peering) 2024-09-17T00:29:04.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:04 smithi099 ceph-mon[56712]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 6 pgs inactive, 6 pgs peering) 2024-09-17T00:29:04.717 INFO:journalctl@ceph.osd.6.smithi099.stdout:Sep 17 00:29:04 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-6[64592]: 2024-09-17T00:29:04.537+0000 7f83a1e08640 -1 Falling back to public interface 2024-09-17T00:29:05.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:05 smithi002 ceph-mon[66732]: pgmap v205: 161 pgs: 33 active+undersized, 20 active+undersized+degraded, 108 active+clean; 585 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 895 B/s rd, 0 op/s; 87/726 objects degraded (11.983%) 2024-09-17T00:29:05.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:05 smithi002 ceph-mon[63804]: pgmap v205: 161 pgs: 33 active+undersized, 20 active+undersized+degraded, 108 active+clean; 585 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 895 B/s rd, 0 op/s; 87/726 objects degraded (11.983%) 2024-09-17T00:29:05.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:05 smithi099 ceph-mon[56712]: pgmap v205: 161 pgs: 33 active+undersized, 20 active+undersized+degraded, 108 active+clean; 585 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 895 B/s rd, 0 op/s; 87/726 objects degraded (11.983%) 2024-09-17T00:29:05.910 INFO:teuthology.orchestra.run.smithi002.stdout:true 2024-09-17T00:29:06.625 INFO:teuthology.orchestra.run.smithi002.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-17T00:29:06.626 INFO:teuthology.orchestra.run.smithi002.stdout:alertmanager.a smithi002 *:9093,9094 running (6m) 114s ago 11m 26.1M - 0.25.0 c8568f914cd2 a21e13b3d677 2024-09-17T00:29:06.626 INFO:teuthology.orchestra.run.smithi002.stdout:grafana.a smithi099 *:3000 running (6m) 38s ago 11m 53.7M - 8.3.5 dad864ee21e9 bafb824a3c08 2024-09-17T00:29:06.626 INFO:teuthology.orchestra.run.smithi002.stdout:iscsi.foo.smithi002.vmouem smithi002 starting - - - - 2024-09-17T00:29:06.626 INFO:teuthology.orchestra.run.smithi002.stdout:mgr.x smithi099 *:8443,9283,8765 running (5m) 38s ago 16m 442M - 19.1.1-330-gc2ddf972 ebb0615149de 6aa2ccef9ca0 2024-09-17T00:29:06.626 INFO:teuthology.orchestra.run.smithi002.stdout:mgr.y smithi002 *:8443,9283,8765 running (6m) 114s ago 18m 506M - 19.1.1-330-gc2ddf972 ebb0615149de 8838897e8449 2024-09-17T00:29:06.626 INFO:teuthology.orchestra.run.smithi002.stdout:mon.a smithi002 running (5m) 114s ago 18m 51.3M 2048M 19.1.1-330-gc2ddf972 ebb0615149de a035def53ef6 2024-09-17T00:29:06.626 INFO:teuthology.orchestra.run.smithi002.stdout:mon.b smithi099 running (4m) 38s ago 16m 41.2M 2048M 19.1.1-330-gc2ddf972 ebb0615149de aab3ab73a850 2024-09-17T00:29:06.626 INFO:teuthology.orchestra.run.smithi002.stdout:mon.c smithi002 running (5m) 114s ago 16m 43.4M 2048M 19.1.1-330-gc2ddf972 ebb0615149de 8a510255ae66 2024-09-17T00:29:06.626 INFO:teuthology.orchestra.run.smithi002.stdout:node-exporter.a smithi002 *:9100 running (7m) 114s ago 12m 14.8M - 1.5.0 0da6a335fe13 1030ae01bb32 2024-09-17T00:29:06.626 INFO:teuthology.orchestra.run.smithi002.stdout:node-exporter.b smithi099 *:9100 running (6m) 38s ago 12m 16.6M - 1.5.0 0da6a335fe13 01fcd35dca8a 2024-09-17T00:29:06.626 INFO:teuthology.orchestra.run.smithi002.stdout:osd.0 smithi002 running (3m) 114s ago 16m 45.8M 2737M 19.1.1-330-gc2ddf972 ebb0615149de 8345719cb6f0 2024-09-17T00:29:06.626 INFO:teuthology.orchestra.run.smithi002.stdout:osd.1 smithi002 running (3m) 114s ago 15m 51.5M 2737M 19.1.1-330-gc2ddf972 ebb0615149de 3dbc23d57c53 2024-09-17T00:29:06.627 INFO:teuthology.orchestra.run.smithi002.stdout:osd.2 smithi002 running (2m) 114s ago 15m 49.3M 2737M 19.1.1-330-gc2ddf972 ebb0615149de d21401b6763b 2024-09-17T00:29:06.627 INFO:teuthology.orchestra.run.smithi002.stdout:osd.3 smithi002 running (119s) 114s ago 14m 12.2M 2737M 19.1.1-330-gc2ddf972 ebb0615149de 980f2a25d096 2024-09-17T00:29:06.627 INFO:teuthology.orchestra.run.smithi002.stdout:osd.4 smithi099 running (80s) 38s ago 14m 47.8M 2992M 19.1.1-330-gc2ddf972 ebb0615149de f644e5be4db8 2024-09-17T00:29:06.627 INFO:teuthology.orchestra.run.smithi002.stdout:osd.5 smithi099 running (42s) 38s ago 14m 11.0M 2992M 19.1.1-330-gc2ddf972 ebb0615149de 2df0d88d16ce 2024-09-17T00:29:06.627 INFO:teuthology.orchestra.run.smithi002.stdout:osd.6 smithi099 starting - - - 2992M 2024-09-17T00:29:06.627 INFO:teuthology.orchestra.run.smithi002.stdout:osd.7 smithi099 running (13m) 38s ago 13m 65.0M 2992M 17.2.0 e1d6a67b021e 63be3a1ed0a1 2024-09-17T00:29:06.627 INFO:teuthology.orchestra.run.smithi002.stdout:prometheus.a smithi099 *:9095 running (5m) 38s ago 12m 67.2M - 2.43.0 a07b618ecd1d bf32c2ae11dc 2024-09-17T00:29:06.627 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.foo.smithi002.uwdhcs smithi002 *:8000 running (11m) 114s ago 11m 76.1M - 17.2.0 e1d6a67b021e 9c35813f459a 2024-09-17T00:29:06.627 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.foo.smithi099.utzfap smithi099 *:8000 running (11m) 38s ago 11m 76.9M - 17.2.0 e1d6a67b021e 34ec989160a6 2024-09-17T00:29:06.627 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.smpl.smithi002.rqnxii smithi002 *:80 running (10m) 114s ago 10m 74.4M - 17.2.0 e1d6a67b021e 0e180d0afb99 2024-09-17T00:29:06.627 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.smpl.smithi099.nzvkpm smithi099 *:80 running (10m) 38s ago 10m 77.5M - 17.2.0 e1d6a67b021e 6a068e82ac72 2024-09-17T00:29:06.779 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:29:06 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: ::ffff:172.21.15.99 - - [17/Sep/2024:00:29:06] "GET /metrics HTTP/1.1" 200 38463 "" "Prometheus/2.43.0" 2024-09-17T00:29:07.056 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-09-17T00:29:07.056 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": { 2024-09-17T00:29:07.056 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 19.1.1-330-gc2ddf972 (c2ddf9721ef254645201cdbabd9c8f49c30a586f) squid (stable)": 3 2024-09-17T00:29:07.056 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-09-17T00:29:07.056 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": { 2024-09-17T00:29:07.056 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 19.1.1-330-gc2ddf972 (c2ddf9721ef254645201cdbabd9c8f49c30a586f) squid (stable)": 2 2024-09-17T00:29:07.057 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-09-17T00:29:07.057 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": { 2024-09-17T00:29:07.057 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-09-17T00:29:07.057 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 19.1.1-330-gc2ddf972 (c2ddf9721ef254645201cdbabd9c8f49c30a586f) squid (stable)": 6 2024-09-17T00:29:07.057 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-09-17T00:29:07.057 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw": { 2024-09-17T00:29:07.057 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-09-17T00:29:07.057 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-09-17T00:29:07.057 INFO:teuthology.orchestra.run.smithi002.stdout: "overall": { 2024-09-17T00:29:07.057 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 5, 2024-09-17T00:29:07.057 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 19.1.1-330-gc2ddf972 (c2ddf9721ef254645201cdbabd9c8f49c30a586f) squid (stable)": 11 2024-09-17T00:29:07.057 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-09-17T00:29:07.057 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-09-17T00:29:07.419 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-09-17T00:29:07.419 INFO:teuthology.orchestra.run.smithi002.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f", 2024-09-17T00:29:07.419 INFO:teuthology.orchestra.run.smithi002.stdout: "in_progress": true, 2024-09-17T00:29:07.419 INFO:teuthology.orchestra.run.smithi002.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-17T00:29:07.420 INFO:teuthology.orchestra.run.smithi002.stdout: "services_complete": [ 2024-09-17T00:29:07.420 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr", 2024-09-17T00:29:07.420 INFO:teuthology.orchestra.run.smithi002.stdout: "mon" 2024-09-17T00:29:07.420 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-09-17T00:29:07.420 INFO:teuthology.orchestra.run.smithi002.stdout: "progress": "12/23 daemons upgraded", 2024-09-17T00:29:07.420 INFO:teuthology.orchestra.run.smithi002.stdout: "message": "Currently upgrading osd daemons", 2024-09-17T00:29:07.420 INFO:teuthology.orchestra.run.smithi002.stdout: "is_paused": false 2024-09-17T00:29:07.420 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-09-17T00:29:07.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:07 smithi099 ceph-mon[56712]: from='client.34794 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:29:07.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:07 smithi099 ceph-mon[56712]: pgmap v206: 161 pgs: 33 active+undersized, 20 active+undersized+degraded, 108 active+clean; 585 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 801 B/s rd, 0 op/s; 87/726 objects degraded (11.983%) 2024-09-17T00:29:07.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:07 smithi099 ceph-mon[56712]: from='client.34800 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:29:07.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:07 smithi099 ceph-mon[56712]: from='client.54533 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:29:07.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:07 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:29:07.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:07 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:29:07.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:07 smithi099 ceph-mon[56712]: from='client.? 172.21.15.2:0/208849525' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:29:07.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:07 smithi002 ceph-mon[63804]: from='client.34794 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:29:07.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:07 smithi002 ceph-mon[63804]: pgmap v206: 161 pgs: 33 active+undersized, 20 active+undersized+degraded, 108 active+clean; 585 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 801 B/s rd, 0 op/s; 87/726 objects degraded (11.983%) 2024-09-17T00:29:07.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:07 smithi002 ceph-mon[63804]: from='client.34800 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:29:07.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:07 smithi002 ceph-mon[63804]: from='client.54533 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:29:07.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:07 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:29:07.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:07 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:29:07.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:07 smithi002 ceph-mon[63804]: from='client.? 172.21.15.2:0/208849525' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:29:07.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:07 smithi002 ceph-mon[66732]: from='client.34794 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:29:07.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:07 smithi002 ceph-mon[66732]: pgmap v206: 161 pgs: 33 active+undersized, 20 active+undersized+degraded, 108 active+clean; 585 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 801 B/s rd, 0 op/s; 87/726 objects degraded (11.983%) 2024-09-17T00:29:07.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:07 smithi002 ceph-mon[66732]: from='client.34800 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:29:07.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:07 smithi002 ceph-mon[66732]: from='client.54533 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:29:07.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:07 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:29:07.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:07 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:29:07.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:07 smithi002 ceph-mon[66732]: from='client.? 172.21.15.2:0/208849525' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:29:07.870 INFO:teuthology.orchestra.run.smithi002.stdout:HEALTH_WARN 1 osds down; Degraded data redundancy: 87/726 objects degraded (11.983%), 20 pgs degraded 2024-09-17T00:29:07.870 INFO:teuthology.orchestra.run.smithi002.stdout:[WRN] OSD_DOWN: 1 osds down 2024-09-17T00:29:07.870 INFO:teuthology.orchestra.run.smithi002.stdout: osd.6 (root=default,host=smithi099) is down 2024-09-17T00:29:07.870 INFO:teuthology.orchestra.run.smithi002.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 87/726 objects degraded (11.983%), 20 pgs degraded 2024-09-17T00:29:07.870 INFO:teuthology.orchestra.run.smithi002.stdout: pg 1.0 is active+undersized+degraded, acting [7,0] 2024-09-17T00:29:07.870 INFO:teuthology.orchestra.run.smithi002.stdout: pg 2.2 is active+undersized+degraded, acting [5,1] 2024-09-17T00:29:07.870 INFO:teuthology.orchestra.run.smithi002.stdout: pg 2.11 is active+undersized+degraded, acting [4,1] 2024-09-17T00:29:07.870 INFO:teuthology.orchestra.run.smithi002.stdout: pg 2.16 is active+undersized+degraded, acting [5,2] 2024-09-17T00:29:07.870 INFO:teuthology.orchestra.run.smithi002.stdout: pg 2.1b is active+undersized+degraded, acting [3,7] 2024-09-17T00:29:07.871 INFO:teuthology.orchestra.run.smithi002.stdout: pg 2.1d is active+undersized+degraded, acting [7,0] 2024-09-17T00:29:07.871 INFO:teuthology.orchestra.run.smithi002.stdout: pg 3.0 is active+undersized+degraded, acting [1,2] 2024-09-17T00:29:07.871 INFO:teuthology.orchestra.run.smithi002.stdout: pg 3.2 is active+undersized+degraded, acting [3,5] 2024-09-17T00:29:07.871 INFO:teuthology.orchestra.run.smithi002.stdout: pg 3.3 is active+undersized+degraded, acting [4,0] 2024-09-17T00:29:07.871 INFO:teuthology.orchestra.run.smithi002.stdout: pg 3.a is active+undersized+degraded, acting [4,1] 2024-09-17T00:29:07.871 INFO:teuthology.orchestra.run.smithi002.stdout: pg 3.c is active+undersized+degraded, acting [5,3] 2024-09-17T00:29:07.871 INFO:teuthology.orchestra.run.smithi002.stdout: pg 3.d is active+undersized+degraded, acting [7,5] 2024-09-17T00:29:07.871 INFO:teuthology.orchestra.run.smithi002.stdout: pg 3.10 is active+undersized+degraded, acting [0,5] 2024-09-17T00:29:07.871 INFO:teuthology.orchestra.run.smithi002.stdout: pg 3.11 is active+undersized+degraded, acting [7,4] 2024-09-17T00:29:07.871 INFO:teuthology.orchestra.run.smithi002.stdout: pg 3.14 is active+undersized+degraded, acting [4,7] 2024-09-17T00:29:07.871 INFO:teuthology.orchestra.run.smithi002.stdout: pg 3.1d is active+undersized+degraded, acting [5,4] 2024-09-17T00:29:07.871 INFO:teuthology.orchestra.run.smithi002.stdout: pg 3.1e is active+undersized+degraded, acting [3,2] 2024-09-17T00:29:07.871 INFO:teuthology.orchestra.run.smithi002.stdout: pg 4.e is active+undersized+degraded, acting [4,1] 2024-09-17T00:29:07.872 INFO:teuthology.orchestra.run.smithi002.stdout: pg 4.1f is active+undersized+degraded, acting [5,1] 2024-09-17T00:29:07.872 INFO:teuthology.orchestra.run.smithi002.stdout: pg 6.c is active+undersized+degraded, acting [3,5] 2024-09-17T00:29:08.467 INFO:journalctl@ceph.osd.6.smithi099.stdout:Sep 17 00:29:08 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-6[64592]: 2024-09-17T00:29:08.075+0000 7f83a1e08640 -1 osd.6 0 read_superblock omap replica is missing. 2024-09-17T00:29:08.467 INFO:journalctl@ceph.osd.6.smithi099.stdout:Sep 17 00:29:08 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-6[64592]: 2024-09-17T00:29:08.111+0000 7f83a1e08640 -1 osd.6 142 log_to_monitors true 2024-09-17T00:29:08.793 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:08 smithi099 ceph-mon[56712]: from='client.54506 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:29:08.793 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:08 smithi099 ceph-mon[56712]: from='client.34818 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:29:08.793 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:08 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:29:08.793 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:08 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:29:08.793 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:08 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:29:08.793 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:08 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:29:08.793 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:08 smithi099 ceph-mon[56712]: from='client.? 172.21.15.2:0/1067393568' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T00:29:08.793 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:08 smithi099 ceph-mon[56712]: from='osd.6 [v2:172.21.15.99:6816/1562011947,v1:172.21.15.99:6817/1562011947]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-17T00:29:08.793 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:08 smithi099 ceph-mon[56712]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-17T00:29:08.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:08 smithi002 ceph-mon[66732]: from='client.54506 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:29:08.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:08 smithi002 ceph-mon[66732]: from='client.34818 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:29:08.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:08 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:29:08.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:08 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:29:08.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:08 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:29:08.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:08 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:29:08.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:08 smithi002 ceph-mon[66732]: from='client.? 172.21.15.2:0/1067393568' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T00:29:08.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:08 smithi002 ceph-mon[66732]: from='osd.6 [v2:172.21.15.99:6816/1562011947,v1:172.21.15.99:6817/1562011947]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-17T00:29:08.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:08 smithi002 ceph-mon[66732]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-17T00:29:08.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:08 smithi002 ceph-mon[63804]: from='client.54506 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:29:08.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:08 smithi002 ceph-mon[63804]: from='client.34818 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:29:08.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:08 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:29:08.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:08 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:29:08.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:08 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:29:08.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:08 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:29:08.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:08 smithi002 ceph-mon[63804]: from='client.? 172.21.15.2:0/1067393568' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T00:29:08.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:08 smithi002 ceph-mon[63804]: from='osd.6 [v2:172.21.15.99:6816/1562011947,v1:172.21.15.99:6817/1562011947]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-17T00:29:08.981 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:08 smithi002 ceph-mon[63804]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-17T00:29:09.147 INFO:journalctl@ceph.osd.6.smithi099.stdout:Sep 17 00:29:08 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-6[64592]: 2024-09-17T00:29:08.794+0000 7f8399bb3640 -1 osd.6 142 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T00:29:09.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:09 smithi099 ceph-mon[56712]: pgmap v207: 161 pgs: 33 active+undersized, 20 active+undersized+degraded, 108 active+clean; 585 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 818 B/s rd, 0 op/s; 87/726 objects degraded (11.983%) 2024-09-17T00:29:09.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:09 smithi099 ceph-mon[56712]: Health check update: Degraded data redundancy: 87/726 objects degraded (11.983%), 20 pgs degraded (PG_DEGRADED) 2024-09-17T00:29:09.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:09 smithi099 ceph-mon[56712]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-17T00:29:09.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:09 smithi099 ceph-mon[56712]: osdmap e145: 8 total, 7 up, 8 in 2024-09-17T00:29:09.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:09 smithi099 ceph-mon[56712]: from='osd.6 [v2:172.21.15.99:6816/1562011947,v1:172.21.15.99:6817/1562011947]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi099", "root=default"]}]: dispatch 2024-09-17T00:29:09.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:09 smithi099 ceph-mon[56712]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi099", "root=default"]}]: dispatch 2024-09-17T00:29:09.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:09 smithi002 ceph-mon[63804]: pgmap v207: 161 pgs: 33 active+undersized, 20 active+undersized+degraded, 108 active+clean; 585 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 818 B/s rd, 0 op/s; 87/726 objects degraded (11.983%) 2024-09-17T00:29:09.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:09 smithi002 ceph-mon[63804]: Health check update: Degraded data redundancy: 87/726 objects degraded (11.983%), 20 pgs degraded (PG_DEGRADED) 2024-09-17T00:29:09.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:09 smithi002 ceph-mon[63804]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-17T00:29:09.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:09 smithi002 ceph-mon[63804]: osdmap e145: 8 total, 7 up, 8 in 2024-09-17T00:29:09.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:09 smithi002 ceph-mon[63804]: from='osd.6 [v2:172.21.15.99:6816/1562011947,v1:172.21.15.99:6817/1562011947]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi099", "root=default"]}]: dispatch 2024-09-17T00:29:09.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:09 smithi002 ceph-mon[63804]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi099", "root=default"]}]: dispatch 2024-09-17T00:29:09.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:09 smithi002 ceph-mon[66732]: pgmap v207: 161 pgs: 33 active+undersized, 20 active+undersized+degraded, 108 active+clean; 585 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 818 B/s rd, 0 op/s; 87/726 objects degraded (11.983%) 2024-09-17T00:29:09.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:09 smithi002 ceph-mon[66732]: Health check update: Degraded data redundancy: 87/726 objects degraded (11.983%), 20 pgs degraded (PG_DEGRADED) 2024-09-17T00:29:09.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:09 smithi002 ceph-mon[66732]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-17T00:29:09.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:09 smithi002 ceph-mon[66732]: osdmap e145: 8 total, 7 up, 8 in 2024-09-17T00:29:09.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:09 smithi002 ceph-mon[66732]: from='osd.6 [v2:172.21.15.99:6816/1562011947,v1:172.21.15.99:6817/1562011947]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi099", "root=default"]}]: dispatch 2024-09-17T00:29:09.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:09 smithi002 ceph-mon[66732]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi099", "root=default"]}]: dispatch 2024-09-17T00:29:10.616 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:10 smithi099 ceph-mon[56712]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-17T00:29:10.616 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:10 smithi099 ceph-mon[56712]: osd.6 [v2:172.21.15.99:6816/1562011947,v1:172.21.15.99:6817/1562011947] boot 2024-09-17T00:29:10.616 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:10 smithi099 ceph-mon[56712]: osdmap e146: 8 total, 8 up, 8 in 2024-09-17T00:29:10.616 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:10 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T00:29:10.801 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:10 smithi002 ceph-mon[63804]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-17T00:29:10.801 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:10 smithi002 ceph-mon[63804]: osd.6 [v2:172.21.15.99:6816/1562011947,v1:172.21.15.99:6817/1562011947] boot 2024-09-17T00:29:10.801 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:10 smithi002 ceph-mon[63804]: osdmap e146: 8 total, 8 up, 8 in 2024-09-17T00:29:10.801 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:10 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T00:29:10.801 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:10 smithi002 ceph-mon[66732]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-17T00:29:10.802 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:10 smithi002 ceph-mon[66732]: osd.6 [v2:172.21.15.99:6816/1562011947,v1:172.21.15.99:6817/1562011947] boot 2024-09-17T00:29:10.802 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:10 smithi002 ceph-mon[66732]: osdmap e146: 8 total, 8 up, 8 in 2024-09-17T00:29:10.802 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:10 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-17T00:29:11.229 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:29:10 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: 2024-09-17T00:29:10.800+0000 7f17efec7640 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (20 PGs are or would become offline) 2024-09-17T00:29:11.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:11 smithi099 ceph-mon[56712]: pgmap v210: 161 pgs: 33 active+undersized, 20 active+undersized+degraded, 108 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 87/726 objects degraded (11.983%) 2024-09-17T00:29:11.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:11 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:29:11.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:11 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:29:11.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:11 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:29:11.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:11 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:29:11.968 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:11 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:29:11.968 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:11 smithi099 ceph-mon[56712]: osdmap e147: 8 total, 8 up, 8 in 2024-09-17T00:29:11.968 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:11 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T00:29:11.968 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:11 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:29:11.968 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:11 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:29:11.968 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:11 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:29:11.968 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:11 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-09-17T00:29:11.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:11 smithi002 ceph-mon[63804]: pgmap v210: 161 pgs: 33 active+undersized, 20 active+undersized+degraded, 108 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 87/726 objects degraded (11.983%) 2024-09-17T00:29:11.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:11 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:29:11.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:11 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:29:11.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:11 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:29:11.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:11 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:29:11.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:11 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:29:11.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:11 smithi002 ceph-mon[63804]: osdmap e147: 8 total, 8 up, 8 in 2024-09-17T00:29:11.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:11 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T00:29:11.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:11 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:29:11.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:11 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:29:11.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:11 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:29:11.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:11 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-09-17T00:29:11.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:11 smithi002 ceph-mon[66732]: pgmap v210: 161 pgs: 33 active+undersized, 20 active+undersized+degraded, 108 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 87/726 objects degraded (11.983%) 2024-09-17T00:29:11.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:11 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:29:11.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:11 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:29:11.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:11 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:29:11.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:11 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:29:11.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:11 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:29:11.981 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:11 smithi002 ceph-mon[66732]: osdmap e147: 8 total, 8 up, 8 in 2024-09-17T00:29:11.981 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:11 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T00:29:11.981 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:11 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:29:11.981 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:11 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:29:11.981 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:11 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:29:11.981 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:11 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-09-17T00:29:12.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:12 smithi099 ceph-mon[56712]: from='mon.1 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-09-17T00:29:12.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:12 smithi099 ceph-mon[56712]: Upgrade: unsafe to stop osd(s) at this time (20 PGs are or would become offline) 2024-09-17T00:29:12.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:12 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:29:12.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:12 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T00:29:12.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:12 smithi002 ceph-mon[63804]: from='mon.1 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-09-17T00:29:12.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:12 smithi002 ceph-mon[63804]: Upgrade: unsafe to stop osd(s) at this time (20 PGs are or would become offline) 2024-09-17T00:29:12.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:12 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:29:12.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:12 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T00:29:12.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:12 smithi002 ceph-mon[66732]: from='mon.1 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-09-17T00:29:12.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:12 smithi002 ceph-mon[66732]: Upgrade: unsafe to stop osd(s) at this time (20 PGs are or would become offline) 2024-09-17T00:29:12.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:12 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:29:12.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:12 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T00:29:13.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:13 smithi099 ceph-mon[56712]: pgmap v212: 161 pgs: 7 peering, 26 active+undersized, 15 active+undersized+degraded, 113 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 62/726 objects degraded (8.540%) 2024-09-17T00:29:13.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:13 smithi002 ceph-mon[66732]: pgmap v212: 161 pgs: 7 peering, 26 active+undersized, 15 active+undersized+degraded, 113 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 62/726 objects degraded (8.540%) 2024-09-17T00:29:13.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:13 smithi002 ceph-mon[63804]: pgmap v212: 161 pgs: 7 peering, 26 active+undersized, 15 active+undersized+degraded, 113 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 62/726 objects degraded (8.540%) 2024-09-17T00:29:14.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:14 smithi099 ceph-mon[56712]: Health check update: Degraded data redundancy: 62/726 objects degraded (8.540%), 15 pgs degraded (PG_DEGRADED) 2024-09-17T00:29:14.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:14 smithi002 ceph-mon[66732]: Health check update: Degraded data redundancy: 62/726 objects degraded (8.540%), 15 pgs degraded (PG_DEGRADED) 2024-09-17T00:29:14.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:14 smithi002 ceph-mon[63804]: Health check update: Degraded data redundancy: 62/726 objects degraded (8.540%), 15 pgs degraded (PG_DEGRADED) 2024-09-17T00:29:15.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:15 smithi099 ceph-mon[56712]: pgmap v213: 161 pgs: 7 peering, 154 active+clean; 585 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-17T00:29:15.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:15 smithi099 ceph-mon[56712]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 62/726 objects degraded (8.540%), 15 pgs degraded) 2024-09-17T00:29:15.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:15 smithi099 ceph-mon[56712]: Cluster is now healthy 2024-09-17T00:29:15.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:15 smithi002 ceph-mon[66732]: pgmap v213: 161 pgs: 7 peering, 154 active+clean; 585 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-17T00:29:15.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:15 smithi002 ceph-mon[66732]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 62/726 objects degraded (8.540%), 15 pgs degraded) 2024-09-17T00:29:15.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:15 smithi002 ceph-mon[66732]: Cluster is now healthy 2024-09-17T00:29:15.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:15 smithi002 ceph-mon[63804]: pgmap v213: 161 pgs: 7 peering, 154 active+clean; 585 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-17T00:29:15.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:15 smithi002 ceph-mon[63804]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 62/726 objects degraded (8.540%), 15 pgs degraded) 2024-09-17T00:29:15.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:15 smithi002 ceph-mon[63804]: Cluster is now healthy 2024-09-17T00:29:16.979 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:29:16 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: ::ffff:172.21.15.99 - - [17/Sep/2024:00:29:16] "GET /metrics HTTP/1.1" 200 38332 "" "Prometheus/2.43.0" 2024-09-17T00:29:17.581 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:17 smithi002 ceph-mon[63804]: pgmap v214: 161 pgs: 161 active+clean; 585 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-17T00:29:17.581 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:17 smithi002 ceph-mon[66732]: pgmap v214: 161 pgs: 161 active+clean; 585 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-17T00:29:17.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:17 smithi099 ceph-mon[56712]: pgmap v214: 161 pgs: 161 active+clean; 585 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-17T00:29:18.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:18 smithi099 ceph-mon[56712]: from='client.54506 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:29:18.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:18 smithi002 ceph-mon[66732]: from='client.54506 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:29:18.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:18 smithi002 ceph-mon[63804]: from='client.54506 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:29:19.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:19 smithi099 ceph-mon[56712]: pgmap v215: 161 pgs: 161 active+clean; 585 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 364 B/s rd, 0 op/s 2024-09-17T00:29:19.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:19 smithi002 ceph-mon[66732]: pgmap v215: 161 pgs: 161 active+clean; 585 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 364 B/s rd, 0 op/s 2024-09-17T00:29:19.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:19 smithi002 ceph-mon[63804]: pgmap v215: 161 pgs: 161 active+clean; 585 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 364 B/s rd, 0 op/s 2024-09-17T00:29:21.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:21 smithi099 ceph-mon[56712]: pgmap v216: 161 pgs: 161 active+clean; 585 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 818 B/s rd, 0 op/s 2024-09-17T00:29:21.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:21 smithi002 ceph-mon[63804]: pgmap v216: 161 pgs: 161 active+clean; 585 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 818 B/s rd, 0 op/s 2024-09-17T00:29:21.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:21 smithi002 ceph-mon[66732]: pgmap v216: 161 pgs: 161 active+clean; 585 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 818 B/s rd, 0 op/s 2024-09-17T00:29:23.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:22 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:29:23.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:22 smithi099 ceph-mon[56712]: pgmap v217: 161 pgs: 161 active+clean; 585 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 716 B/s rd, 0 op/s 2024-09-17T00:29:23.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:22 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:29:23.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:22 smithi002 ceph-mon[66732]: pgmap v217: 161 pgs: 161 active+clean; 585 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 716 B/s rd, 0 op/s 2024-09-17T00:29:23.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:22 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:29:23.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:22 smithi002 ceph-mon[63804]: pgmap v217: 161 pgs: 161 active+clean; 585 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 716 B/s rd, 0 op/s 2024-09-17T00:29:25.920 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:25 smithi099 ceph-mon[56712]: pgmap v218: 161 pgs: 161 active+clean; 585 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-17T00:29:25.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:25 smithi002 ceph-mon[66732]: pgmap v218: 161 pgs: 161 active+clean; 585 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-17T00:29:25.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:25 smithi002 ceph-mon[63804]: pgmap v218: 161 pgs: 161 active+clean; 585 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-17T00:29:26.858 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:26 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-09-17T00:29:26.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:26 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-09-17T00:29:26.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:26 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-09-17T00:29:26.980 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:29:26 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: ::ffff:172.21.15.99 - - [17/Sep/2024:00:29:26] "GET /metrics HTTP/1.1" 200 38332 "" "Prometheus/2.43.0" 2024-09-17T00:29:27.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:27 smithi002 ceph-mon[63804]: from='mon.1 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-09-17T00:29:27.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:27 smithi002 ceph-mon[63804]: Upgrade: osd.7 is safe to restart 2024-09-17T00:29:27.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:27 smithi002 ceph-mon[63804]: pgmap v219: 161 pgs: 161 active+clean; 585 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:29:27.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:27 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:29:27.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:27 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T00:29:27.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:27 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:29:27.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:27 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-09-17T00:29:27.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:27 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:29:27.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:27 smithi002 ceph-mon[66732]: from='mon.1 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-09-17T00:29:27.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:27 smithi002 ceph-mon[66732]: Upgrade: osd.7 is safe to restart 2024-09-17T00:29:27.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:27 smithi002 ceph-mon[66732]: pgmap v219: 161 pgs: 161 active+clean; 585 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:29:27.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:27 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:29:27.731 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:27 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T00:29:27.731 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:27 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:29:27.731 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:27 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-09-17T00:29:27.731 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:27 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:29:27.855 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:27 smithi099 ceph-mon[56712]: from='mon.1 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-09-17T00:29:27.855 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:27 smithi099 ceph-mon[56712]: Upgrade: osd.7 is safe to restart 2024-09-17T00:29:27.856 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:27 smithi099 ceph-mon[56712]: pgmap v219: 161 pgs: 161 active+clean; 585 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:29:27.856 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:27 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:29:27.856 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:27 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T00:29:27.856 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:27 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:29:27.856 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:27 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-09-17T00:29:27.856 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:27 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:29:28.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:28 smithi099 ceph-mon[56712]: from='client.54506 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:29:28.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:28 smithi099 ceph-mon[56712]: Upgrade: Updating osd.7 2024-09-17T00:29:28.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:28 smithi099 ceph-mon[56712]: Deploying daemon osd.7 on smithi099 2024-09-17T00:29:28.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:28 smithi002 ceph-mon[63804]: from='client.54506 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:29:28.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:28 smithi002 ceph-mon[63804]: Upgrade: Updating osd.7 2024-09-17T00:29:28.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:28 smithi002 ceph-mon[63804]: Deploying daemon osd.7 on smithi099 2024-09-17T00:29:28.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:28 smithi002 ceph-mon[66732]: from='client.54506 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:29:28.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:28 smithi002 ceph-mon[66732]: Upgrade: Updating osd.7 2024-09-17T00:29:28.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:28 smithi002 ceph-mon[66732]: Deploying daemon osd.7 on smithi099 2024-09-17T00:29:29.713 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:29 smithi099 ceph-mon[56712]: pgmap v220: 161 pgs: 161 active+clean; 585 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:29:29.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:29 smithi002 ceph-mon[63804]: pgmap v220: 161 pgs: 161 active+clean; 585 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:29:29.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:29 smithi002 ceph-mon[66732]: pgmap v220: 161 pgs: 161 active+clean; 585 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:29:30.668 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:29:30 smithi099 systemd[1]: Stopping Ceph osd.7 for 3aad27a2-7489-11ef-bceb-c7b262605968... 2024-09-17T00:29:30.967 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:29:30 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-7[36631]: 2024-09-17T00:29:30.670+0000 7efc9c7db700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-17T00:29:30.967 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:29:30 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-7[36631]: 2024-09-17T00:29:30.670+0000 7efc9c7db700 -1 osd.7 147 *** Got signal Terminated *** 2024-09-17T00:29:30.967 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:29:30 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-7[36631]: 2024-09-17T00:29:30.670+0000 7efc9c7db700 -1 osd.7 147 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-17T00:29:31.895 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:31 smithi099 ceph-mon[56712]: pgmap v221: 161 pgs: 161 active+clean; 585 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:29:31.895 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:31 smithi099 ceph-mon[56712]: osd.7 marked itself down and dead 2024-09-17T00:29:31.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:31 smithi002 ceph-mon[66732]: pgmap v221: 161 pgs: 161 active+clean; 585 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:29:31.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:31 smithi002 ceph-mon[66732]: osd.7 marked itself down and dead 2024-09-17T00:29:31.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:31 smithi002 ceph-mon[63804]: pgmap v221: 161 pgs: 161 active+clean; 585 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:29:31.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:31 smithi002 ceph-mon[63804]: osd.7 marked itself down and dead 2024-09-17T00:29:32.169 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:29:31 smithi099 podman[66456]: 2024-09-17 00:29:31.897513793 +0000 UTC m=+1.349808674 container died 63be3a1ed0a14e7dd41d9fa9513ea735b009efb252e70655456ae30e91492a30 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-7, com.redhat.component=centos-stream-container, vendor=Red Hat, Inc., version=8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-type=git, distribution-scope=public, maintainer=Guillaume Abrioux , io.buildah.version=1.19.8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.expose-services=, io.openshift.tags=base centos centos-stream, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_BRANCH=HEAD, build-date=2022-05-03T08:36:31.336870, GIT_CLEAN=True, release=754, io.k8s.display-name=CentOS Stream 8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, RELEASE=HEAD, name=centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, CEPH_POINT_RELEASE=-17.2.0, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, ceph=True) 2024-09-17T00:29:32.467 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:29:32 smithi099 podman[66456]: 2024-09-17 00:29:32.345643256 +0000 UTC m=+1.797938183 container cleanup 63be3a1ed0a14e7dd41d9fa9513ea735b009efb252e70655456ae30e91492a30 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-7, name=centos-stream, version=8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, ceph=True, architecture=x86_64, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-type=git, build-date=2022-05-03T08:36:31.336870, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=CentOS Stream 8, io.openshift.tags=base centos centos-stream, CEPH_POINT_RELEASE=-17.2.0, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.19.8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.expose-services=, com.redhat.component=centos-stream-container, distribution-scope=public, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_BRANCH=HEAD, GIT_CLEAN=True, release=754, vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754) 2024-09-17T00:29:32.467 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:29:32 smithi099 bash[66456]: ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-7 2024-09-17T00:29:32.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:32 smithi099 ceph-mon[56712]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-17T00:29:32.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:32 smithi099 ceph-mon[56712]: Health check failed: all OSDs are running squid or later but require_osd_release < squid (OSD_UPGRADE_FINISHED) 2024-09-17T00:29:32.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:32 smithi099 ceph-mon[56712]: osdmap e148: 8 total, 7 up, 8 in 2024-09-17T00:29:32.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:32 smithi099 ceph-mon[56712]: osdmap e149: 8 total, 7 up, 8 in 2024-09-17T00:29:32.967 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:29:32 smithi099 podman[66472]: 2024-09-17 00:29:32.555926553 +0000 UTC m=+0.650654508 container remove 63be3a1ed0a14e7dd41d9fa9513ea735b009efb252e70655456ae30e91492a30 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-7, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, CEPH_POINT_RELEASE=-17.2.0, architecture=x86_64, io.openshift.expose-services=, release=754, GIT_BRANCH=HEAD, build-date=2022-05-03T08:36:31.336870, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.component=centos-stream-container, name=centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, distribution-scope=public, io.k8s.display-name=CentOS Stream 8, io.openshift.tags=base centos centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.19.8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, RELEASE=HEAD, GIT_CLEAN=True, vcs-type=git, vendor=Red Hat, Inc., version=8, maintainer=Guillaume Abrioux ) 2024-09-17T00:29:32.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:32 smithi002 ceph-mon[66732]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-17T00:29:32.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:32 smithi002 ceph-mon[66732]: Health check failed: all OSDs are running squid or later but require_osd_release < squid (OSD_UPGRADE_FINISHED) 2024-09-17T00:29:32.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:32 smithi002 ceph-mon[66732]: osdmap e148: 8 total, 7 up, 8 in 2024-09-17T00:29:32.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:32 smithi002 ceph-mon[66732]: osdmap e149: 8 total, 7 up, 8 in 2024-09-17T00:29:32.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:32 smithi002 ceph-mon[63804]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-17T00:29:32.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:32 smithi002 ceph-mon[63804]: Health check failed: all OSDs are running squid or later but require_osd_release < squid (OSD_UPGRADE_FINISHED) 2024-09-17T00:29:32.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:32 smithi002 ceph-mon[63804]: osdmap e148: 8 total, 7 up, 8 in 2024-09-17T00:29:32.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:32 smithi002 ceph-mon[63804]: osdmap e149: 8 total, 7 up, 8 in 2024-09-17T00:29:33.273 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:29:32 smithi099 podman[66560]: 2024-09-17 00:29:32.878815432 +0000 UTC m=+0.016139107 image pull ebb0615149def4a3179c82c0cc47af75808300abb9f1142788e5e605a7ac0e5b quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f 2024-09-17T00:29:33.273 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:29:32 smithi099 podman[66560]: 2024-09-17 00:29:32.99851352 +0000 UTC m=+0.135837180 container create 25a0aa99bc16926e97344e52bb05db8302b8f950fcbc573c915e665eee44a7c8 (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-7-deactivate, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=squid-c2ddf97, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-09-17T00:29:33.273 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:29:33 smithi099 podman[66560]: 2024-09-17 00:29:33.200953896 +0000 UTC m=+0.338277564 container init 25a0aa99bc16926e97344e52bb05db8302b8f950fcbc573c915e665eee44a7c8 (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-7-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=squid-c2ddf97, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS) 2024-09-17T00:29:33.273 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:29:33 smithi099 podman[66560]: 2024-09-17 00:29:33.204766588 +0000 UTC m=+0.342090248 container start 25a0aa99bc16926e97344e52bb05db8302b8f950fcbc573c915e665eee44a7c8 (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-7-deactivate, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, CEPH_POINT_RELEASE=, RELEASE=squid-c2ddf97, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731) 2024-09-17T00:29:33.274 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:29:33 smithi099 podman[66560]: 2024-09-17 00:29:33.275936301 +0000 UTC m=+0.413259963 container attach 25a0aa99bc16926e97344e52bb05db8302b8f950fcbc573c915e665eee44a7c8 (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-7-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, ceph=True, CEPH_POINT_RELEASE=, RELEASE=squid-c2ddf97, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-09-17T00:29:33.576 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:29:33 smithi099 podman[66560]: 2024-09-17 00:29:33.392798496 +0000 UTC m=+0.530122167 container died 25a0aa99bc16926e97344e52bb05db8302b8f950fcbc573c915e665eee44a7c8 (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-7-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=squid-c2ddf97, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_CLEAN=True) 2024-09-17T00:29:33.926 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:33 smithi099 ceph-mon[56712]: pgmap v223: 161 pgs: 23 stale+active+clean, 138 active+clean; 585 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-17T00:29:33.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:33 smithi002 ceph-mon[63804]: pgmap v223: 161 pgs: 23 stale+active+clean, 138 active+clean; 585 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-17T00:29:33.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:33 smithi002 ceph-mon[66732]: pgmap v223: 161 pgs: 23 stale+active+clean, 138 active+clean; 585 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-17T00:29:34.217 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:29:33 smithi099 podman[66583]: 2024-09-17 00:29:33.929594987 +0000 UTC m=+0.530934824 container remove 25a0aa99bc16926e97344e52bb05db8302b8f950fcbc573c915e665eee44a7c8 (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-7-deactivate, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=squid-c2ddf97, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD) 2024-09-17T00:29:34.217 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:29:33 smithi099 systemd[1]: ceph-3aad27a2-7489-11ef-bceb-c7b262605968@osd.7.service: Deactivated successfully. 2024-09-17T00:29:34.217 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:29:33 smithi099 systemd[1]: Stopped Ceph osd.7 for 3aad27a2-7489-11ef-bceb-c7b262605968. 2024-09-17T00:29:34.217 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:29:33 smithi099 systemd[1]: ceph-3aad27a2-7489-11ef-bceb-c7b262605968@osd.7.service: Consumed 10.641s CPU time. 2024-09-17T00:29:34.217 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:29:34 smithi099 systemd[1]: Starting Ceph osd.7 for 3aad27a2-7489-11ef-bceb-c7b262605968... 2024-09-17T00:29:34.629 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:29:34 smithi099 podman[66682]: 2024-09-17 00:29:34.261433082 +0000 UTC m=+0.015753827 image pull ebb0615149def4a3179c82c0cc47af75808300abb9f1142788e5e605a7ac0e5b quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f 2024-09-17T00:29:34.629 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:29:34 smithi099 podman[66682]: 2024-09-17 00:29:34.381332114 +0000 UTC m=+0.135652900 container create 9635011ec0e10f073733a44964b9cf0743fde06d3b6291fc6c9d26acc95486c7 (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-7-activate, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, ceph=True, RELEASE=squid-c2ddf97, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0) 2024-09-17T00:29:34.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:34 smithi099 ceph-mon[56712]: pgmap v225: 161 pgs: 1 activating+undersized, 20 active+undersized, 10 stale+active+clean, 14 active+undersized+degraded, 116 active+clean; 585 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 63/726 objects degraded (8.678%) 2024-09-17T00:29:34.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:34 smithi099 ceph-mon[56712]: Health check failed: Degraded data redundancy: 63/726 objects degraded (8.678%), 14 pgs degraded (PG_DEGRADED) 2024-09-17T00:29:34.967 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:29:34 smithi099 podman[66682]: 2024-09-17 00:29:34.642031632 +0000 UTC m=+0.396352383 container init 9635011ec0e10f073733a44964b9cf0743fde06d3b6291fc6c9d26acc95486c7 (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-7-activate, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, ceph=True, RELEASE=squid-c2ddf97, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-09-17T00:29:34.967 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:29:34 smithi099 podman[66682]: 2024-09-17 00:29:34.645357765 +0000 UTC m=+0.399678519 container start 9635011ec0e10f073733a44964b9cf0743fde06d3b6291fc6c9d26acc95486c7 (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-7-activate, RELEASE=squid-c2ddf97, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2) 2024-09-17T00:29:34.967 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:29:34 smithi099 podman[66682]: 2024-09-17 00:29:34.71718284 +0000 UTC m=+0.471503596 container attach 9635011ec0e10f073733a44964b9cf0743fde06d3b6291fc6c9d26acc95486c7 (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-7-activate, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-c2ddf97, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2) 2024-09-17T00:29:34.968 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:29:34 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-7-activate[66698]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-17T00:29:34.968 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:29:34 smithi099 bash[66682]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-17T00:29:34.968 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:29:34 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-7-activate[66698]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-17T00:29:34.968 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:29:34 smithi099 bash[66682]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-17T00:29:34.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:34 smithi002 ceph-mon[63804]: pgmap v225: 161 pgs: 1 activating+undersized, 20 active+undersized, 10 stale+active+clean, 14 active+undersized+degraded, 116 active+clean; 585 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 63/726 objects degraded (8.678%) 2024-09-17T00:29:34.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:34 smithi002 ceph-mon[63804]: Health check failed: Degraded data redundancy: 63/726 objects degraded (8.678%), 14 pgs degraded (PG_DEGRADED) 2024-09-17T00:29:34.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:34 smithi002 ceph-mon[66732]: pgmap v225: 161 pgs: 1 activating+undersized, 20 active+undersized, 10 stale+active+clean, 14 active+undersized+degraded, 116 active+clean; 585 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 63/726 objects degraded (8.678%) 2024-09-17T00:29:34.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:34 smithi002 ceph-mon[66732]: Health check failed: Degraded data redundancy: 63/726 objects degraded (8.678%), 14 pgs degraded (PG_DEGRADED) 2024-09-17T00:29:35.523 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:29:35 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-7-activate[66698]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-09-17T00:29:35.523 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:29:35 smithi099 bash[66682]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-09-17T00:29:35.523 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:29:35 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-7-activate[66698]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-7 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-09-17T00:29:35.523 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:29:35 smithi099 bash[66682]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-7 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-09-17T00:29:35.523 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:29:35 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-7-activate[66698]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-09-17T00:29:35.523 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:29:35 smithi099 bash[66682]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-09-17T00:29:35.826 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:29:35 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-7-activate[66698]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-09-17T00:29:35.826 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:29:35 smithi099 bash[66682]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-09-17T00:29:35.826 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:29:35 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-7-activate[66698]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-7/block 2024-09-17T00:29:35.826 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:29:35 smithi099 bash[66682]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-7/block 2024-09-17T00:29:35.826 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:29:35 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-7-activate[66698]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-09-17T00:29:35.826 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:29:35 smithi099 bash[66682]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-09-17T00:29:35.826 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:29:35 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-7-activate[66698]: --> ceph-volume raw activate successful for osd ID: 7 2024-09-17T00:29:35.827 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:29:35 smithi099 bash[66682]: --> ceph-volume raw activate successful for osd ID: 7 2024-09-17T00:29:35.827 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:29:35 smithi099 podman[66682]: 2024-09-17 00:29:35.553984699 +0000 UTC m=+1.308305444 container died 9635011ec0e10f073733a44964b9cf0743fde06d3b6291fc6c9d26acc95486c7 (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-7-activate, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, RELEASE=squid-c2ddf97, org.label-schema.license=GPLv2, io.buildah.version=1.37.2) 2024-09-17T00:29:36.467 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:29:36 smithi099 podman[66829]: 2024-09-17 00:29:36.120859024 +0000 UTC m=+0.560870593 container remove 9635011ec0e10f073733a44964b9cf0743fde06d3b6291fc6c9d26acc95486c7 (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-7-activate, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, RELEASE=squid-c2ddf97, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-17T00:29:36.467 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:29:36 smithi099 podman[66888]: 2024-09-17 00:29:36.231479775 +0000 UTC m=+0.016158084 image pull ebb0615149def4a3179c82c0cc47af75808300abb9f1142788e5e605a7ac0e5b quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f 2024-09-17T00:29:36.467 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:29:36 smithi099 podman[66888]: 2024-09-17 00:29:36.363812161 +0000 UTC m=+0.148490464 container create 9821b8012a7e191d67981f3f04e6e48f8034fcc427cbcd8e8a5ac6030270c3ba (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-7, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-c2ddf97, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-09-17T00:29:36.967 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:29:36 smithi099 podman[66888]: 2024-09-17 00:29:36.607780103 +0000 UTC m=+0.392458407 container init 9821b8012a7e191d67981f3f04e6e48f8034fcc427cbcd8e8a5ac6030270c3ba (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-7, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=squid-c2ddf97, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-09-17T00:29:36.967 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:29:36 smithi099 podman[66888]: 2024-09-17 00:29:36.610993344 +0000 UTC m=+0.395671648 container start 9821b8012a7e191d67981f3f04e6e48f8034fcc427cbcd8e8a5ac6030270c3ba (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-7, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-c2ddf97, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-09-17T00:29:36.967 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:29:36 smithi099 ceph-osd[66906]: -- 172.21.15.99:0/3318856026 <== mon.2 v2:172.21.15.99:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55820b80fa40 con 0x55820c5d6000 2024-09-17T00:29:36.967 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:29:36 smithi099 bash[66888]: 9821b8012a7e191d67981f3f04e6e48f8034fcc427cbcd8e8a5ac6030270c3ba 2024-09-17T00:29:36.967 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:29:36 smithi099 systemd[1]: Started Ceph osd.7 for 3aad27a2-7489-11ef-bceb-c7b262605968. 2024-09-17T00:29:36.979 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:29:36 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: ::ffff:172.21.15.99 - - [17/Sep/2024:00:29:36] "GET /metrics HTTP/1.1" 200 38478 "" "Prometheus/2.43.0" 2024-09-17T00:29:37.330 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:37 smithi002 ceph-mon[63804]: pgmap v226: 161 pgs: 1 activating+undersized, 32 active+undersized, 4 stale+active+clean, 18 active+undersized+degraded, 106 active+clean; 585 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 80/726 objects degraded (11.019%) 2024-09-17T00:29:37.330 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:37 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:29:37.330 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:37 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:29:37.331 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:37 smithi002 ceph-mon[66732]: pgmap v226: 161 pgs: 1 activating+undersized, 32 active+undersized, 4 stale+active+clean, 18 active+undersized+degraded, 106 active+clean; 585 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 80/726 objects degraded (11.019%) 2024-09-17T00:29:37.331 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:37 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:29:37.331 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:37 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:29:37.566 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:37 smithi099 ceph-mon[56712]: pgmap v226: 161 pgs: 1 activating+undersized, 32 active+undersized, 4 stale+active+clean, 18 active+undersized+degraded, 106 active+clean; 585 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 80/726 objects degraded (11.019%) 2024-09-17T00:29:37.566 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:37 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:29:37.567 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:37 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:29:38.237 INFO:teuthology.orchestra.run.smithi002.stdout:true 2024-09-17T00:29:38.277 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:29:38 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-7[66902]: 2024-09-17T00:29:38.010+0000 7f5a59013640 -1 Falling back to public interface 2024-09-17T00:29:38.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:38 smithi099 ceph-mon[56712]: from='client.54506 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:29:38.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:38 smithi002 ceph-mon[66732]: from='client.54506 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:29:38.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:38 smithi002 ceph-mon[63804]: from='client.54506 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:29:38.928 INFO:teuthology.orchestra.run.smithi002.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-17T00:29:38.929 INFO:teuthology.orchestra.run.smithi002.stdout:alertmanager.a smithi002 *:9093,9094 running (7m) 31s ago 12m 26.3M - 0.25.0 c8568f914cd2 a21e13b3d677 2024-09-17T00:29:38.929 INFO:teuthology.orchestra.run.smithi002.stdout:grafana.a smithi099 *:3000 running (7m) 32s ago 12m 54.0M - 8.3.5 dad864ee21e9 bafb824a3c08 2024-09-17T00:29:38.929 INFO:teuthology.orchestra.run.smithi002.stdout:iscsi.foo.smithi002.vmouem smithi002 running (62s) 31s ago 11m 48.3M - 3.5 e1d6a67b021e e7f7bcd32d4e 2024-09-17T00:29:38.929 INFO:teuthology.orchestra.run.smithi002.stdout:mgr.x smithi099 *:8443,9283,8765 running (6m) 32s ago 16m 442M - 19.1.1-330-gc2ddf972 ebb0615149de 6aa2ccef9ca0 2024-09-17T00:29:38.929 INFO:teuthology.orchestra.run.smithi002.stdout:mgr.y smithi002 *:8443,9283,8765 running (6m) 31s ago 18m 509M - 19.1.1-330-gc2ddf972 ebb0615149de 8838897e8449 2024-09-17T00:29:38.929 INFO:teuthology.orchestra.run.smithi002.stdout:mon.a smithi002 running (6m) 31s ago 18m 56.9M 2048M 19.1.1-330-gc2ddf972 ebb0615149de a035def53ef6 2024-09-17T00:29:38.930 INFO:teuthology.orchestra.run.smithi002.stdout:mon.b smithi099 running (5m) 32s ago 17m 41.5M 2048M 19.1.1-330-gc2ddf972 ebb0615149de aab3ab73a850 2024-09-17T00:29:38.930 INFO:teuthology.orchestra.run.smithi002.stdout:mon.c smithi002 running (5m) 31s ago 17m 46.9M 2048M 19.1.1-330-gc2ddf972 ebb0615149de 8a510255ae66 2024-09-17T00:29:38.930 INFO:teuthology.orchestra.run.smithi002.stdout:node-exporter.a smithi002 *:9100 running (7m) 31s ago 13m 15.4M - 1.5.0 0da6a335fe13 1030ae01bb32 2024-09-17T00:29:38.930 INFO:teuthology.orchestra.run.smithi002.stdout:node-exporter.b smithi099 *:9100 running (7m) 32s ago 13m 16.7M - 1.5.0 0da6a335fe13 01fcd35dca8a 2024-09-17T00:29:38.930 INFO:teuthology.orchestra.run.smithi002.stdout:osd.0 smithi002 running (4m) 31s ago 16m 51.4M 2737M 19.1.1-330-gc2ddf972 ebb0615149de 8345719cb6f0 2024-09-17T00:29:38.930 INFO:teuthology.orchestra.run.smithi002.stdout:osd.1 smithi002 running (3m) 31s ago 16m 57.7M 2737M 19.1.1-330-gc2ddf972 ebb0615149de 3dbc23d57c53 2024-09-17T00:29:38.931 INFO:teuthology.orchestra.run.smithi002.stdout:osd.2 smithi002 running (3m) 31s ago 15m 53.1M 2737M 19.1.1-330-gc2ddf972 ebb0615149de d21401b6763b 2024-09-17T00:29:38.931 INFO:teuthology.orchestra.run.smithi002.stdout:osd.3 smithi002 running (2m) 31s ago 15m 51.4M 2737M 19.1.1-330-gc2ddf972 ebb0615149de 980f2a25d096 2024-09-17T00:29:38.931 INFO:teuthology.orchestra.run.smithi002.stdout:osd.4 smithi099 running (112s) 32s ago 15m 48.6M 2992M 19.1.1-330-gc2ddf972 ebb0615149de f644e5be4db8 2024-09-17T00:29:38.931 INFO:teuthology.orchestra.run.smithi002.stdout:osd.5 smithi099 running (75s) 32s ago 14m 50.7M 2992M 19.1.1-330-gc2ddf972 ebb0615149de 2df0d88d16ce 2024-09-17T00:29:38.931 INFO:teuthology.orchestra.run.smithi002.stdout:osd.6 smithi099 running (36s) 32s ago 14m 11.6M 2992M 19.1.1-330-gc2ddf972 ebb0615149de 7462bdbdf29b 2024-09-17T00:29:38.931 INFO:teuthology.orchestra.run.smithi002.stdout:osd.7 smithi099 starting - - - 2992M 2024-09-17T00:29:38.932 INFO:teuthology.orchestra.run.smithi002.stdout:prometheus.a smithi099 *:9095 running (6m) 32s ago 13m 67.3M - 2.43.0 a07b618ecd1d bf32c2ae11dc 2024-09-17T00:29:38.932 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.foo.smithi002.uwdhcs smithi002 *:8000 running (11m) 31s ago 11m 76.6M - 17.2.0 e1d6a67b021e 9c35813f459a 2024-09-17T00:29:38.932 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.foo.smithi099.utzfap smithi099 *:8000 running (11m) 32s ago 11m 77.1M - 17.2.0 e1d6a67b021e 34ec989160a6 2024-09-17T00:29:38.932 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.smpl.smithi002.rqnxii smithi002 *:80 running (11m) 31s ago 11m 75.1M - 17.2.0 e1d6a67b021e 0e180d0afb99 2024-09-17T00:29:38.932 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.smpl.smithi099.nzvkpm smithi099 *:80 running (11m) 32s ago 11m 77.7M - 17.2.0 e1d6a67b021e 6a068e82ac72 2024-09-17T00:29:39.343 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-09-17T00:29:39.344 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": { 2024-09-17T00:29:39.344 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 19.1.1-330-gc2ddf972 (c2ddf9721ef254645201cdbabd9c8f49c30a586f) squid (stable)": 3 2024-09-17T00:29:39.344 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-09-17T00:29:39.344 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": { 2024-09-17T00:29:39.344 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 19.1.1-330-gc2ddf972 (c2ddf9721ef254645201cdbabd9c8f49c30a586f) squid (stable)": 2 2024-09-17T00:29:39.344 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-09-17T00:29:39.344 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": { 2024-09-17T00:29:39.344 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 19.1.1-330-gc2ddf972 (c2ddf9721ef254645201cdbabd9c8f49c30a586f) squid (stable)": 7 2024-09-17T00:29:39.344 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-09-17T00:29:39.344 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw": { 2024-09-17T00:29:39.345 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-09-17T00:29:39.345 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-09-17T00:29:39.345 INFO:teuthology.orchestra.run.smithi002.stdout: "overall": { 2024-09-17T00:29:39.345 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4, 2024-09-17T00:29:39.345 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 19.1.1-330-gc2ddf972 (c2ddf9721ef254645201cdbabd9c8f49c30a586f) squid (stable)": 12 2024-09-17T00:29:39.345 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-09-17T00:29:39.345 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-09-17T00:29:39.692 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-09-17T00:29:39.692 INFO:teuthology.orchestra.run.smithi002.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f", 2024-09-17T00:29:39.692 INFO:teuthology.orchestra.run.smithi002.stdout: "in_progress": true, 2024-09-17T00:29:39.692 INFO:teuthology.orchestra.run.smithi002.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-17T00:29:39.692 INFO:teuthology.orchestra.run.smithi002.stdout: "services_complete": [ 2024-09-17T00:29:39.692 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr", 2024-09-17T00:29:39.692 INFO:teuthology.orchestra.run.smithi002.stdout: "mon" 2024-09-17T00:29:39.692 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-09-17T00:29:39.692 INFO:teuthology.orchestra.run.smithi002.stdout: "progress": "12/23 daemons upgraded", 2024-09-17T00:29:39.693 INFO:teuthology.orchestra.run.smithi002.stdout: "message": "Currently upgrading osd daemons", 2024-09-17T00:29:39.693 INFO:teuthology.orchestra.run.smithi002.stdout: "is_paused": false 2024-09-17T00:29:39.693 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-09-17T00:29:39.693 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:39 smithi002 ceph-mon[63804]: pgmap v227: 161 pgs: 45 active+undersized, 24 active+undersized+degraded, 92 active+clean; 585 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 94/726 objects degraded (12.948%) 2024-09-17T00:29:39.693 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:39 smithi002 ceph-mon[63804]: from='client.34833 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:29:39.693 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:39 smithi002 ceph-mon[63804]: from='client.44746 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:29:39.693 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:39 smithi002 ceph-mon[66732]: pgmap v227: 161 pgs: 45 active+undersized, 24 active+undersized+degraded, 92 active+clean; 585 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 94/726 objects degraded (12.948%) 2024-09-17T00:29:39.693 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:39 smithi002 ceph-mon[66732]: from='client.34833 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:29:39.693 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:39 smithi002 ceph-mon[66732]: from='client.44746 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:29:39.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:39 smithi099 ceph-mon[56712]: pgmap v227: 161 pgs: 45 active+undersized, 24 active+undersized+degraded, 92 active+clean; 585 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 94/726 objects degraded (12.948%) 2024-09-17T00:29:39.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:39 smithi099 ceph-mon[56712]: from='client.34833 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:29:39.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:39 smithi099 ceph-mon[56712]: from='client.44746 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:29:40.105 INFO:teuthology.orchestra.run.smithi002.stdout:HEALTH_WARN 1 osds down; all OSDs are running squid or later but require_osd_release < squid; Degraded data redundancy: 94/726 objects degraded (12.948%), 24 pgs degraded 2024-09-17T00:29:40.105 INFO:teuthology.orchestra.run.smithi002.stdout:[WRN] OSD_DOWN: 1 osds down 2024-09-17T00:29:40.105 INFO:teuthology.orchestra.run.smithi002.stdout: osd.7 (root=default,host=smithi099) is down 2024-09-17T00:29:40.105 INFO:teuthology.orchestra.run.smithi002.stdout:[WRN] OSD_UPGRADE_FINISHED: all OSDs are running squid or later but require_osd_release < squid 2024-09-17T00:29:40.105 INFO:teuthology.orchestra.run.smithi002.stdout: all OSDs are running squid or later but require_osd_release < squid 2024-09-17T00:29:40.105 INFO:teuthology.orchestra.run.smithi002.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 94/726 objects degraded (12.948%), 24 pgs degraded 2024-09-17T00:29:40.105 INFO:teuthology.orchestra.run.smithi002.stdout: pg 1.0 is active+undersized+degraded, acting [0,6] 2024-09-17T00:29:40.105 INFO:teuthology.orchestra.run.smithi002.stdout: pg 2.9 is active+undersized+degraded, acting [1,3] 2024-09-17T00:29:40.105 INFO:teuthology.orchestra.run.smithi002.stdout: pg 2.a is active+undersized+degraded, acting [1,3] 2024-09-17T00:29:40.105 INFO:teuthology.orchestra.run.smithi002.stdout: pg 2.e is active+undersized+degraded, acting [2,3] 2024-09-17T00:29:40.105 INFO:teuthology.orchestra.run.smithi002.stdout: pg 2.f is active+undersized+degraded, acting [4,0] 2024-09-17T00:29:40.106 INFO:teuthology.orchestra.run.smithi002.stdout: pg 2.1b is active+undersized+degraded, acting [3,6] 2024-09-17T00:29:40.106 INFO:teuthology.orchestra.run.smithi002.stdout: pg 2.1d is active+undersized+degraded, acting [6,0] 2024-09-17T00:29:40.106 INFO:teuthology.orchestra.run.smithi002.stdout: pg 3.7 is active+undersized+degraded, acting [3,0] 2024-09-17T00:29:40.106 INFO:teuthology.orchestra.run.smithi002.stdout: pg 3.8 is active+undersized+degraded, acting [3,1] 2024-09-17T00:29:40.106 INFO:teuthology.orchestra.run.smithi002.stdout: pg 3.9 is active+undersized+degraded, acting [4,2] 2024-09-17T00:29:40.106 INFO:teuthology.orchestra.run.smithi002.stdout: pg 3.d is active+undersized+degraded, acting [5,6] 2024-09-17T00:29:40.106 INFO:teuthology.orchestra.run.smithi002.stdout: pg 3.e is active+undersized+degraded, acting [4,1] 2024-09-17T00:29:40.106 INFO:teuthology.orchestra.run.smithi002.stdout: pg 3.f is active+undersized+degraded, acting [4,0] 2024-09-17T00:29:40.106 INFO:teuthology.orchestra.run.smithi002.stdout: pg 3.11 is active+undersized+degraded, acting [4,6] 2024-09-17T00:29:40.106 INFO:teuthology.orchestra.run.smithi002.stdout: pg 3.12 is active+undersized+degraded, acting [0,3] 2024-09-17T00:29:40.106 INFO:teuthology.orchestra.run.smithi002.stdout: pg 3.13 is active+undersized+degraded, acting [4,2] 2024-09-17T00:29:40.106 INFO:teuthology.orchestra.run.smithi002.stdout: pg 3.14 is active+undersized+degraded, acting [4,6] 2024-09-17T00:29:40.106 INFO:teuthology.orchestra.run.smithi002.stdout: pg 3.15 is active+undersized+degraded, acting [3,4] 2024-09-17T00:29:40.106 INFO:teuthology.orchestra.run.smithi002.stdout: pg 3.16 is active+undersized+degraded, acting [5,1] 2024-09-17T00:29:40.107 INFO:teuthology.orchestra.run.smithi002.stdout: pg 3.1b is active+undersized+degraded, acting [0,4] 2024-09-17T00:29:40.107 INFO:teuthology.orchestra.run.smithi002.stdout: pg 4.0 is active+undersized+degraded, acting [3,0] 2024-09-17T00:29:40.107 INFO:teuthology.orchestra.run.smithi002.stdout: pg 4.3 is active+undersized+degraded, acting [0,5] 2024-09-17T00:29:40.107 INFO:teuthology.orchestra.run.smithi002.stdout: pg 4.14 is active+undersized+degraded, acting [3,1] 2024-09-17T00:29:40.107 INFO:teuthology.orchestra.run.smithi002.stdout: pg 4.15 is active+undersized+degraded, acting [5,3] 2024-09-17T00:29:40.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:40 smithi002 ceph-mon[66732]: from='client.34845 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:29:40.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:40 smithi002 ceph-mon[66732]: from='client.? 172.21.15.2:0/1495334338' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:29:40.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:40 smithi002 ceph-mon[66732]: from='client.? 172.21.15.2:0/163044449' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T00:29:40.480 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:40 smithi002 ceph-mon[63804]: from='client.34845 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:29:40.480 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:40 smithi002 ceph-mon[63804]: from='client.? 172.21.15.2:0/1495334338' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:29:40.480 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:40 smithi002 ceph-mon[63804]: from='client.? 172.21.15.2:0/163044449' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T00:29:40.548 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:40 smithi099 ceph-mon[56712]: from='client.34845 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:29:40.548 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:40 smithi099 ceph-mon[56712]: from='client.? 172.21.15.2:0/1495334338' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:29:40.548 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:40 smithi099 ceph-mon[56712]: from='client.? 172.21.15.2:0/163044449' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T00:29:41.281 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:29:41 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-7[66902]: 2024-09-17T00:29:41.017+0000 7f5a59013640 -1 osd.7 0 read_superblock omap replica is missing. 2024-09-17T00:29:41.281 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:29:41 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-7[66902]: 2024-09-17T00:29:41.048+0000 7f5a59013640 -1 osd.7 147 log_to_monitors true 2024-09-17T00:29:41.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:41 smithi099 ceph-mon[56712]: from='client.44764 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:29:41.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:41 smithi099 ceph-mon[56712]: pgmap v228: 161 pgs: 45 active+undersized, 24 active+undersized+degraded, 92 active+clean; 585 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 94/726 objects degraded (12.948%) 2024-09-17T00:29:41.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:41 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:29:41.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:41 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:29:41.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:41 smithi099 ceph-mon[56712]: from='osd.7 [v2:172.21.15.99:6824/619773135,v1:172.21.15.99:6825/619773135]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-17T00:29:41.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:41 smithi099 ceph-mon[56712]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-17T00:29:41.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:41 smithi002 ceph-mon[66732]: from='client.44764 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:29:41.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:41 smithi002 ceph-mon[66732]: pgmap v228: 161 pgs: 45 active+undersized, 24 active+undersized+degraded, 92 active+clean; 585 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 94/726 objects degraded (12.948%) 2024-09-17T00:29:41.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:41 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:29:41.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:41 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:29:41.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:41 smithi002 ceph-mon[66732]: from='osd.7 [v2:172.21.15.99:6824/619773135,v1:172.21.15.99:6825/619773135]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-17T00:29:41.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:41 smithi002 ceph-mon[66732]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-17T00:29:41.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:41 smithi002 ceph-mon[63804]: from='client.44764 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:29:41.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:41 smithi002 ceph-mon[63804]: pgmap v228: 161 pgs: 45 active+undersized, 24 active+undersized+degraded, 92 active+clean; 585 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 94/726 objects degraded (12.948%) 2024-09-17T00:29:41.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:41 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:29:41.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:41 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:29:41.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:41 smithi002 ceph-mon[63804]: from='osd.7 [v2:172.21.15.99:6824/619773135,v1:172.21.15.99:6825/619773135]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-17T00:29:41.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:41 smithi002 ceph-mon[63804]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-17T00:29:42.596 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:29:42 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-7[66902]: 2024-09-17T00:29:42.456+0000 7f5a4cdb6640 -1 osd.7 147 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-17T00:29:42.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:42 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:29:42.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:42 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:29:42.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:42 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:29:42.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:42 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T00:29:42.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:42 smithi099 ceph-mon[56712]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-17T00:29:42.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:42 smithi099 ceph-mon[56712]: osdmap e150: 8 total, 7 up, 8 in 2024-09-17T00:29:42.968 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:42 smithi099 ceph-mon[56712]: from='osd.7 [v2:172.21.15.99:6824/619773135,v1:172.21.15.99:6825/619773135]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi099", "root=default"]}]: dispatch 2024-09-17T00:29:42.968 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:42 smithi099 ceph-mon[56712]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi099", "root=default"]}]: dispatch 2024-09-17T00:29:42.968 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:42 smithi099 ceph-mon[56712]: from='osd.7 ' entity='osd.7' 2024-09-17T00:29:42.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:42 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:29:42.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:42 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:29:42.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:42 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:29:42.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:42 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T00:29:42.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:42 smithi002 ceph-mon[66732]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-17T00:29:42.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:42 smithi002 ceph-mon[66732]: osdmap e150: 8 total, 7 up, 8 in 2024-09-17T00:29:42.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:42 smithi002 ceph-mon[66732]: from='osd.7 [v2:172.21.15.99:6824/619773135,v1:172.21.15.99:6825/619773135]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi099", "root=default"]}]: dispatch 2024-09-17T00:29:42.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:42 smithi002 ceph-mon[66732]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi099", "root=default"]}]: dispatch 2024-09-17T00:29:42.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:42 smithi002 ceph-mon[66732]: from='osd.7 ' entity='osd.7' 2024-09-17T00:29:42.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:42 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:29:42.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:42 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:29:42.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:42 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:29:42.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:42 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T00:29:42.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:42 smithi002 ceph-mon[63804]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-17T00:29:42.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:42 smithi002 ceph-mon[63804]: osdmap e150: 8 total, 7 up, 8 in 2024-09-17T00:29:42.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:42 smithi002 ceph-mon[63804]: from='osd.7 [v2:172.21.15.99:6824/619773135,v1:172.21.15.99:6825/619773135]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi099", "root=default"]}]: dispatch 2024-09-17T00:29:42.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:42 smithi002 ceph-mon[63804]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi099", "root=default"]}]: dispatch 2024-09-17T00:29:42.981 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:42 smithi002 ceph-mon[63804]: from='osd.7 ' entity='osd.7' 2024-09-17T00:29:43.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:43 smithi099 ceph-mon[56712]: pgmap v230: 161 pgs: 45 active+undersized, 24 active+undersized+degraded, 92 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 94/726 objects degraded (12.948%) 2024-09-17T00:29:43.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:43 smithi099 ceph-mon[56712]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-17T00:29:43.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:43 smithi099 ceph-mon[56712]: osd.7 [v2:172.21.15.99:6824/619773135,v1:172.21.15.99:6825/619773135] boot 2024-09-17T00:29:43.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:43 smithi099 ceph-mon[56712]: osdmap e151: 8 total, 8 up, 8 in 2024-09-17T00:29:43.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:43 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T00:29:43.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:43 smithi099 ceph-mon[56712]: Health check update: Degraded data redundancy: 94/726 objects degraded (12.948%), 24 pgs degraded (PG_DEGRADED) 2024-09-17T00:29:43.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:43 smithi002 ceph-mon[66732]: pgmap v230: 161 pgs: 45 active+undersized, 24 active+undersized+degraded, 92 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 94/726 objects degraded (12.948%) 2024-09-17T00:29:43.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:43 smithi002 ceph-mon[66732]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-17T00:29:43.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:43 smithi002 ceph-mon[66732]: osd.7 [v2:172.21.15.99:6824/619773135,v1:172.21.15.99:6825/619773135] boot 2024-09-17T00:29:43.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:43 smithi002 ceph-mon[66732]: osdmap e151: 8 total, 8 up, 8 in 2024-09-17T00:29:43.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:43 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T00:29:43.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:43 smithi002 ceph-mon[66732]: Health check update: Degraded data redundancy: 94/726 objects degraded (12.948%), 24 pgs degraded (PG_DEGRADED) 2024-09-17T00:29:43.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:43 smithi002 ceph-mon[63804]: pgmap v230: 161 pgs: 45 active+undersized, 24 active+undersized+degraded, 92 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 94/726 objects degraded (12.948%) 2024-09-17T00:29:43.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:43 smithi002 ceph-mon[63804]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-17T00:29:43.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:43 smithi002 ceph-mon[63804]: osd.7 [v2:172.21.15.99:6824/619773135,v1:172.21.15.99:6825/619773135] boot 2024-09-17T00:29:43.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:43 smithi002 ceph-mon[63804]: osdmap e151: 8 total, 8 up, 8 in 2024-09-17T00:29:43.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:43 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-17T00:29:43.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:43 smithi002 ceph-mon[63804]: Health check update: Degraded data redundancy: 94/726 objects degraded (12.948%), 24 pgs degraded (PG_DEGRADED) 2024-09-17T00:29:45.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:45 smithi099 ceph-mon[56712]: pgmap v232: 161 pgs: 18 peering, 32 active+undersized, 19 active+undersized+degraded, 92 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 76/726 objects degraded (10.468%) 2024-09-17T00:29:45.718 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:45 smithi099 ceph-mon[56712]: osdmap e152: 8 total, 8 up, 8 in 2024-09-17T00:29:45.718 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:45 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:29:45.718 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:45 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:29:45.718 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:45 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:29:45.718 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:45 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:29:45.718 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:45 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:29:45.718 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:45 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T00:29:45.718 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:45 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:29:45.718 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:45 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:29:45.718 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:45 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:29:45.718 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:45 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:29:45.718 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:45 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:29:45.718 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:45 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-09-17T00:29:45.719 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:45 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-09-17T00:29:45.719 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:45 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-09-17T00:29:45.719 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:45 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-09-17T00:29:45.719 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:45 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-09-17T00:29:45.719 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:45 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-09-17T00:29:45.719 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:45 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-09-17T00:29:45.719 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:45 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-09-17T00:29:45.719 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:45 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-09-17T00:29:45.719 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:45 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-09-17T00:29:45.719 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:45 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-09-17T00:29:45.719 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:45 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-09-17T00:29:45.719 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:45 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-09-17T00:29:45.719 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:45 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-09-17T00:29:45.719 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:45 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-09-17T00:29:45.719 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:45 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-09-17T00:29:45.720 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:45 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-09-17T00:29:45.720 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:45 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-09-17T00:29:45.720 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:45 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]: dispatch 2024-09-17T00:29:45.720 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:45 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]: dispatch 2024-09-17T00:29:45.720 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:45 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]': finished 2024-09-17T00:29:45.720 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:45 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]: dispatch 2024-09-17T00:29:45.720 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:45 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]: dispatch 2024-09-17T00:29:45.720 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:45 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]': finished 2024-09-17T00:29:45.720 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:45 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-09-17T00:29:45.720 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:45 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-09-17T00:29:45.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:45 smithi002 ceph-mon[66732]: pgmap v232: 161 pgs: 18 peering, 32 active+undersized, 19 active+undersized+degraded, 92 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 76/726 objects degraded (10.468%) 2024-09-17T00:29:45.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:45 smithi002 ceph-mon[66732]: osdmap e152: 8 total, 8 up, 8 in 2024-09-17T00:29:45.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:45 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:29:45.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:45 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:29:45.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:45 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:29:45.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:45 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:29:45.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:45 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:29:45.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:45 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T00:29:45.731 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:45 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:29:45.731 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:45 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:29:45.731 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:45 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:29:45.731 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:45 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:29:45.731 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:45 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:29:45.731 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:45 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-09-17T00:29:45.731 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:45 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-09-17T00:29:45.731 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:45 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-09-17T00:29:45.731 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:45 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-09-17T00:29:45.731 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:45 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-09-17T00:29:45.731 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:45 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-09-17T00:29:45.731 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:45 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-09-17T00:29:45.731 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:45 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-09-17T00:29:45.731 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:45 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-09-17T00:29:45.732 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:45 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-09-17T00:29:45.732 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:45 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-09-17T00:29:45.732 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:45 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-09-17T00:29:45.732 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:45 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-09-17T00:29:45.732 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:45 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-09-17T00:29:45.732 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:45 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-09-17T00:29:45.732 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:45 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-09-17T00:29:45.732 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:45 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-09-17T00:29:45.732 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:45 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-09-17T00:29:45.732 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:45 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]: dispatch 2024-09-17T00:29:45.732 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:45 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]: dispatch 2024-09-17T00:29:45.733 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:45 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]': finished 2024-09-17T00:29:45.733 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:45 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]: dispatch 2024-09-17T00:29:45.733 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:45 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]: dispatch 2024-09-17T00:29:45.733 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:45 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]': finished 2024-09-17T00:29:45.733 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:45 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-09-17T00:29:45.733 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:45 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-09-17T00:29:45.734 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:45 smithi002 ceph-mon[63804]: pgmap v232: 161 pgs: 18 peering, 32 active+undersized, 19 active+undersized+degraded, 92 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 76/726 objects degraded (10.468%) 2024-09-17T00:29:45.734 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:45 smithi002 ceph-mon[63804]: osdmap e152: 8 total, 8 up, 8 in 2024-09-17T00:29:45.734 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:45 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:29:45.734 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:45 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:29:45.735 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:45 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:29:45.735 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:45 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:29:45.735 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:45 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:29:45.735 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:45 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T00:29:45.735 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:45 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:29:45.735 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:45 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:29:45.735 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:45 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:29:45.735 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:45 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:29:45.735 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:45 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:29:45.735 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:45 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-09-17T00:29:45.735 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:45 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-09-17T00:29:45.735 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:45 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-09-17T00:29:45.735 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:45 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-09-17T00:29:45.735 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:45 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-09-17T00:29:45.736 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:45 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-09-17T00:29:45.736 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:45 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-09-17T00:29:45.736 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:45 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-09-17T00:29:45.736 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:45 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-09-17T00:29:45.736 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:45 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-09-17T00:29:45.736 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:45 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-09-17T00:29:45.736 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:45 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-09-17T00:29:45.736 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:45 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-09-17T00:29:45.736 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:45 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-09-17T00:29:45.736 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:45 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-09-17T00:29:45.736 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:45 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-09-17T00:29:45.736 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:45 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-09-17T00:29:45.736 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:45 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-09-17T00:29:45.736 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:45 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]: dispatch 2024-09-17T00:29:45.737 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:45 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]: dispatch 2024-09-17T00:29:45.737 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:45 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]': finished 2024-09-17T00:29:45.737 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:45 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]: dispatch 2024-09-17T00:29:45.737 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:45 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]: dispatch 2024-09-17T00:29:45.737 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:45 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]': finished 2024-09-17T00:29:45.737 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:45 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-09-17T00:29:45.737 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:45 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-09-17T00:29:46.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:46 smithi099 ceph-mon[56712]: Upgrade: Setting container_image for all osd 2024-09-17T00:29:46.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:46 smithi099 ceph-mon[56712]: Upgrade: Setting require_osd_release to 19 squid 2024-09-17T00:29:46.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:46 smithi099 ceph-mon[56712]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running squid or later but require_osd_release < squid) 2024-09-17T00:29:46.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:46 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' cmd='[{"prefix": "osd require-osd-release", "release": "squid"}]': finished 2024-09-17T00:29:46.718 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:46 smithi099 ceph-mon[56712]: osdmap e153: 8 total, 8 up, 8 in 2024-09-17T00:29:46.718 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:46 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:29:46.718 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:46 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:29:46.729 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:29:46 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: ::ffff:172.21.15.99 - - [17/Sep/2024:00:29:46] "GET /metrics HTTP/1.1" 200 38560 "" "Prometheus/2.43.0" 2024-09-17T00:29:46.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:46 smithi002 ceph-mon[66732]: Upgrade: Setting container_image for all osd 2024-09-17T00:29:46.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:46 smithi002 ceph-mon[66732]: Upgrade: Setting require_osd_release to 19 squid 2024-09-17T00:29:46.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:46 smithi002 ceph-mon[66732]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running squid or later but require_osd_release < squid) 2024-09-17T00:29:46.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:46 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' cmd='[{"prefix": "osd require-osd-release", "release": "squid"}]': finished 2024-09-17T00:29:46.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:46 smithi002 ceph-mon[66732]: osdmap e153: 8 total, 8 up, 8 in 2024-09-17T00:29:46.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:46 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:29:46.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:46 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:29:46.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:46 smithi002 ceph-mon[63804]: Upgrade: Setting container_image for all osd 2024-09-17T00:29:46.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:46 smithi002 ceph-mon[63804]: Upgrade: Setting require_osd_release to 19 squid 2024-09-17T00:29:46.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:46 smithi002 ceph-mon[63804]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running squid or later but require_osd_release < squid) 2024-09-17T00:29:46.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:46 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' cmd='[{"prefix": "osd require-osd-release", "release": "squid"}]': finished 2024-09-17T00:29:46.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:46 smithi002 ceph-mon[63804]: osdmap e153: 8 total, 8 up, 8 in 2024-09-17T00:29:46.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:46 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:29:46.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:46 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:29:47.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:47 smithi099 ceph-mon[56712]: Upgrade: Setting container_image for all mds 2024-09-17T00:29:47.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:47 smithi099 ceph-mon[56712]: pgmap v235: 161 pgs: 18 peering, 25 active+undersized, 16 active+undersized+degraded, 102 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 67/726 objects degraded (9.229%) 2024-09-17T00:29:47.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:47 smithi002 ceph-mon[66732]: Upgrade: Setting container_image for all mds 2024-09-17T00:29:47.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:47 smithi002 ceph-mon[66732]: pgmap v235: 161 pgs: 18 peering, 25 active+undersized, 16 active+undersized+degraded, 102 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 67/726 objects degraded (9.229%) 2024-09-17T00:29:47.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:47 smithi002 ceph-mon[63804]: Upgrade: Setting container_image for all mds 2024-09-17T00:29:47.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:47 smithi002 ceph-mon[63804]: pgmap v235: 161 pgs: 18 peering, 25 active+undersized, 16 active+undersized+degraded, 102 active+clean; 585 KiB data, 293 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 67/726 objects degraded (9.229%) 2024-09-17T00:29:48.939 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:48 smithi002 ceph-mon[63804]: from='client.54506 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:29:48.939 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:48 smithi002 ceph-mon[63804]: Upgrade: Updating rgw.foo.smithi002.uwdhcs (1/4) 2024-09-17T00:29:48.939 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:48 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:29:48.939 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:48 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi002.uwdhcs", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T00:29:48.939 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:48 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi002.uwdhcs", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T00:29:48.939 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:48 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:29:48.939 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:48 smithi002 ceph-mon[63804]: Deploying daemon rgw.foo.smithi002.uwdhcs on smithi002 2024-09-17T00:29:48.939 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:48 smithi002 ceph-mon[63804]: Health check update: Degraded data redundancy: 67/726 objects degraded (9.229%), 16 pgs degraded (PG_DEGRADED) 2024-09-17T00:29:48.940 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:48 smithi002 ceph-mon[66732]: from='client.54506 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:29:48.940 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:48 smithi002 ceph-mon[66732]: Upgrade: Updating rgw.foo.smithi002.uwdhcs (1/4) 2024-09-17T00:29:48.940 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:48 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:29:48.940 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:48 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi002.uwdhcs", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T00:29:48.940 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:48 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi002.uwdhcs", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T00:29:48.940 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:48 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:29:48.940 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:48 smithi002 ceph-mon[66732]: Deploying daemon rgw.foo.smithi002.uwdhcs on smithi002 2024-09-17T00:29:48.940 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:48 smithi002 ceph-mon[66732]: Health check update: Degraded data redundancy: 67/726 objects degraded (9.229%), 16 pgs degraded (PG_DEGRADED) 2024-09-17T00:29:48.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:48 smithi099 ceph-mon[56712]: from='client.54506 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:29:48.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:48 smithi099 ceph-mon[56712]: Upgrade: Updating rgw.foo.smithi002.uwdhcs (1/4) 2024-09-17T00:29:48.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:48 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:29:48.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:48 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi002.uwdhcs", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T00:29:48.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:48 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi002.uwdhcs", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T00:29:48.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:48 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:29:48.968 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:48 smithi099 ceph-mon[56712]: Deploying daemon rgw.foo.smithi002.uwdhcs on smithi002 2024-09-17T00:29:48.968 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:48 smithi099 ceph-mon[56712]: Health check update: Degraded data redundancy: 67/726 objects degraded (9.229%), 16 pgs degraded (PG_DEGRADED) 2024-09-17T00:29:49.870 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:49 smithi002 ceph-mon[66732]: pgmap v236: 161 pgs: 18 peering, 143 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:29:49.870 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:49 smithi002 ceph-mon[66732]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 67/726 objects degraded (9.229%), 16 pgs degraded) 2024-09-17T00:29:49.870 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:49 smithi002 ceph-mon[66732]: Cluster is now healthy 2024-09-17T00:29:49.871 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:49 smithi002 ceph-mon[63804]: pgmap v236: 161 pgs: 18 peering, 143 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:29:49.871 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:49 smithi002 ceph-mon[63804]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 67/726 objects degraded (9.229%), 16 pgs degraded) 2024-09-17T00:29:49.871 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:49 smithi002 ceph-mon[63804]: Cluster is now healthy 2024-09-17T00:29:49.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:49 smithi099 ceph-mon[56712]: pgmap v236: 161 pgs: 18 peering, 143 active+clean; 585 KiB data, 258 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:29:49.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:49 smithi099 ceph-mon[56712]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 67/726 objects degraded (9.229%), 16 pgs degraded) 2024-09-17T00:29:49.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:49 smithi099 ceph-mon[56712]: Cluster is now healthy 2024-09-17T00:29:51.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:51 smithi099 ceph-mon[56712]: pgmap v237: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-17T00:29:51.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:51 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:29:51.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:51 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:29:51.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:51 smithi002 ceph-mon[63804]: pgmap v237: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-17T00:29:51.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:51 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:29:51.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:51 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:29:51.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:51 smithi002 ceph-mon[66732]: pgmap v237: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-17T00:29:51.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:51 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:29:51.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:51 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:29:53.658 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:53 smithi002 ceph-mon[66732]: pgmap v238: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 31 op/s 2024-09-17T00:29:53.658 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:53 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:29:53.658 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:53 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi002.rqnxii", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T00:29:53.659 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:53 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi002.rqnxii", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T00:29:53.659 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:53 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:29:53.659 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:53 smithi002 ceph-mon[63804]: pgmap v238: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 31 op/s 2024-09-17T00:29:53.659 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:53 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:29:53.659 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:53 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi002.rqnxii", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T00:29:53.659 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:53 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi002.rqnxii", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T00:29:53.659 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:53 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:29:53.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:53 smithi099 ceph-mon[56712]: pgmap v238: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 31 op/s 2024-09-17T00:29:53.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:53 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:29:53.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:53 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi002.rqnxii", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T00:29:53.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:53 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi002.rqnxii", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T00:29:53.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:53 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:29:54.824 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:54 smithi002 ceph-mon[63804]: Upgrade: Updating rgw.smpl.smithi002.rqnxii (2/4) 2024-09-17T00:29:54.824 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:54 smithi002 ceph-mon[63804]: Deploying daemon rgw.smpl.smithi002.rqnxii on smithi002 2024-09-17T00:29:54.824 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:54 smithi002 ceph-mon[66732]: Upgrade: Updating rgw.smpl.smithi002.rqnxii (2/4) 2024-09-17T00:29:54.824 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:54 smithi002 ceph-mon[66732]: Deploying daemon rgw.smpl.smithi002.rqnxii on smithi002 2024-09-17T00:29:54.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:54 smithi099 ceph-mon[56712]: Upgrade: Updating rgw.smpl.smithi002.rqnxii (2/4) 2024-09-17T00:29:54.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:54 smithi099 ceph-mon[56712]: Deploying daemon rgw.smpl.smithi002.rqnxii on smithi002 2024-09-17T00:29:55.728 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:55 smithi002 ceph-mon[66732]: pgmap v239: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 0 B/s wr, 85 op/s 2024-09-17T00:29:55.728 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:55 smithi002 ceph-mon[63804]: pgmap v239: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 0 B/s wr, 85 op/s 2024-09-17T00:29:55.868 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:55 smithi099 ceph-mon[56712]: pgmap v239: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 0 B/s wr, 85 op/s 2024-09-17T00:29:56.729 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:29:56 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: ::ffff:172.21.15.99 - - [17/Sep/2024:00:29:56] "GET /metrics HTTP/1.1" 200 38560 "" "Prometheus/2.43.0" 2024-09-17T00:29:57.815 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:57 smithi099 ceph-mon[56712]: pgmap v240: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 0 B/s wr, 101 op/s 2024-09-17T00:29:57.815 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:57 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:29:57.815 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:57 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T00:29:57.815 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:57 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:29:57.815 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:57 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:29:57.816 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:57 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:29:57.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:57 smithi002 ceph-mon[63804]: pgmap v240: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 0 B/s wr, 101 op/s 2024-09-17T00:29:57.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:57 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:29:57.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:57 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T00:29:57.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:57 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:29:57.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:57 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:29:57.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:57 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:29:57.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:57 smithi002 ceph-mon[66732]: pgmap v240: 161 pgs: 161 active+clean; 585 KiB data, 266 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 0 B/s wr, 101 op/s 2024-09-17T00:29:57.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:57 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:29:57.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:57 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T00:29:57.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:57 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:29:57.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:57 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:29:57.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:57 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:29:58.894 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:58 smithi099 ceph-mon[56712]: from='client.54506 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:29:58.894 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:58 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:29:58.894 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:58 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi099.utzfap", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T00:29:58.894 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:58 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi099.utzfap", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T00:29:58.894 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:58 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:29:58.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:58 smithi002 ceph-mon[66732]: from='client.54506 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:29:58.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:58 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:29:58.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:58 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi099.utzfap", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T00:29:58.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:58 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi099.utzfap", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T00:29:58.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:58 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:29:58.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:58 smithi002 ceph-mon[63804]: from='client.54506 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:29:58.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:58 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:29:58.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:58 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi099.utzfap", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T00:29:58.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:58 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi099.utzfap", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T00:29:58.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:58 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:29:59.871 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:59 smithi099 ceph-mon[56712]: pgmap v241: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 91 KiB/s rd, 85 B/s wr, 144 op/s 2024-09-17T00:29:59.871 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:59 smithi099 ceph-mon[56712]: Upgrade: Updating rgw.foo.smithi099.utzfap (3/4) 2024-09-17T00:29:59.871 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:29:59 smithi099 ceph-mon[56712]: Deploying daemon rgw.foo.smithi099.utzfap on smithi099 2024-09-17T00:29:59.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:59 smithi002 ceph-mon[66732]: pgmap v241: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 91 KiB/s rd, 85 B/s wr, 144 op/s 2024-09-17T00:29:59.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:59 smithi002 ceph-mon[66732]: Upgrade: Updating rgw.foo.smithi099.utzfap (3/4) 2024-09-17T00:29:59.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:29:59 smithi002 ceph-mon[66732]: Deploying daemon rgw.foo.smithi099.utzfap on smithi099 2024-09-17T00:29:59.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:59 smithi002 ceph-mon[63804]: pgmap v241: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 91 KiB/s rd, 85 B/s wr, 144 op/s 2024-09-17T00:29:59.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:59 smithi002 ceph-mon[63804]: Upgrade: Updating rgw.foo.smithi099.utzfap (3/4) 2024-09-17T00:29:59.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:29:59 smithi002 ceph-mon[63804]: Deploying daemon rgw.foo.smithi099.utzfap on smithi099 2024-09-17T00:30:00.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:30:00 smithi099 ceph-mon[56712]: overall HEALTH_OK 2024-09-17T00:30:00.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:30:00 smithi002 ceph-mon[66732]: overall HEALTH_OK 2024-09-17T00:30:00.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:30:00 smithi002 ceph-mon[63804]: overall HEALTH_OK 2024-09-17T00:30:01.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:30:01 smithi099 ceph-mon[56712]: pgmap v242: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 110 KiB/s rd, 85 B/s wr, 173 op/s 2024-09-17T00:30:01.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:30:01 smithi002 ceph-mon[63804]: pgmap v242: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 110 KiB/s rd, 85 B/s wr, 173 op/s 2024-09-17T00:30:01.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:30:01 smithi002 ceph-mon[66732]: pgmap v242: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 110 KiB/s rd, 85 B/s wr, 173 op/s 2024-09-17T00:30:03.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:30:03 smithi099 ceph-mon[56712]: pgmap v243: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 157 KiB/s rd, 170 B/s wr, 248 op/s 2024-09-17T00:30:03.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:30:03 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:30:03.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:30:03 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:30:03.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:30:03 smithi002 ceph-mon[66732]: pgmap v243: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 157 KiB/s rd, 170 B/s wr, 248 op/s 2024-09-17T00:30:03.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:30:03 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:30:03.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:30:03 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:30:03.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:30:03 smithi002 ceph-mon[63804]: pgmap v243: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 157 KiB/s rd, 170 B/s wr, 248 op/s 2024-09-17T00:30:03.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:30:03 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:30:03.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:30:03 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:30:05.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:30:04 smithi099 ceph-mon[56712]: Upgrade: Updating rgw.smpl.smithi099.nzvkpm (4/4) 2024-09-17T00:30:05.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:30:04 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:30:05.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:30:04 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi099.nzvkpm", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T00:30:05.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:30:04 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi099.nzvkpm", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T00:30:05.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:30:04 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:30:05.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:30:04 smithi099 ceph-mon[56712]: Deploying daemon rgw.smpl.smithi099.nzvkpm on smithi099 2024-09-17T00:30:05.218 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:30:04 smithi099 ceph-mon[56712]: pgmap v244: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 172 KiB/s rd, 170 B/s wr, 272 op/s 2024-09-17T00:30:05.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:30:04 smithi002 ceph-mon[66732]: Upgrade: Updating rgw.smpl.smithi099.nzvkpm (4/4) 2024-09-17T00:30:05.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:30:04 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:30:05.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:30:04 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi099.nzvkpm", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T00:30:05.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:30:04 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi099.nzvkpm", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T00:30:05.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:30:04 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:30:05.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:30:04 smithi002 ceph-mon[66732]: Deploying daemon rgw.smpl.smithi099.nzvkpm on smithi099 2024-09-17T00:30:05.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:30:04 smithi002 ceph-mon[66732]: pgmap v244: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 172 KiB/s rd, 170 B/s wr, 272 op/s 2024-09-17T00:30:05.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:30:04 smithi002 ceph-mon[63804]: Upgrade: Updating rgw.smpl.smithi099.nzvkpm (4/4) 2024-09-17T00:30:05.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:30:04 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:30:05.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:30:04 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi099.nzvkpm", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T00:30:05.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:30:04 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi099.nzvkpm", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-17T00:30:05.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:30:04 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:30:05.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:30:04 smithi002 ceph-mon[63804]: Deploying daemon rgw.smpl.smithi099.nzvkpm on smithi099 2024-09-17T00:30:05.230 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:30:04 smithi002 ceph-mon[63804]: pgmap v244: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 172 KiB/s rd, 170 B/s wr, 272 op/s 2024-09-17T00:30:06.979 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:30:06 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: ::ffff:172.21.15.99 - - [17/Sep/2024:00:30:06] "GET /metrics HTTP/1.1" 200 38562 "" "Prometheus/2.43.0" 2024-09-17T00:30:07.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:30:07 smithi099 ceph-mon[56712]: pgmap v245: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 152 KiB/s rd, 170 B/s wr, 240 op/s 2024-09-17T00:30:07.732 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:30:07 smithi002 ceph-mon[66732]: pgmap v245: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 152 KiB/s rd, 170 B/s wr, 240 op/s 2024-09-17T00:30:07.733 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:30:07 smithi002 ceph-mon[63804]: pgmap v245: 161 pgs: 161 active+clean; 585 KiB data, 274 MiB used, 715 GiB / 715 GiB avail; 152 KiB/s rd, 170 B/s wr, 240 op/s 2024-09-17T00:30:08.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:30:08 smithi002 ceph-mon[66732]: from='client.54506 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:30:08.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:30:08 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:30:08.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:30:08 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:30:08.480 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:30:08 smithi002 ceph-mon[63804]: from='client.54506 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:30:08.480 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:30:08 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:30:08.480 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:30:08 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:30:08.634 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:30:08 smithi099 ceph-mon[56712]: from='client.54506 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:30:08.634 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:30:08 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:30:08.634 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:30:08 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:30:09.530 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:30:09 smithi002 ceph-mon[66732]: pgmap v246: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 181 KiB/s rd, 255 B/s wr, 285 op/s 2024-09-17T00:30:09.531 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:30:09 smithi002 ceph-mon[63804]: pgmap v246: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 181 KiB/s rd, 255 B/s wr, 285 op/s 2024-09-17T00:30:09.545 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:30:09 smithi099 ceph-mon[56712]: pgmap v246: 161 pgs: 161 active+clean; 585 KiB data, 278 MiB used, 715 GiB / 715 GiB avail; 181 KiB/s rd, 255 B/s wr, 285 op/s 2024-09-17T00:30:10.474 INFO:teuthology.orchestra.run.smithi002.stdout:true 2024-09-17T00:30:11.182 INFO:teuthology.orchestra.run.smithi002.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-17T00:30:11.182 INFO:teuthology.orchestra.run.smithi002.stdout:alertmanager.a smithi002 *:9093,9094 running (8m) 63s ago 12m 26.3M - 0.25.0 c8568f914cd2 a21e13b3d677 2024-09-17T00:30:11.182 INFO:teuthology.orchestra.run.smithi002.stdout:grafana.a smithi099 *:3000 running (7m) 30s ago 12m 54.0M - 8.3.5 dad864ee21e9 bafb824a3c08 2024-09-17T00:30:11.182 INFO:teuthology.orchestra.run.smithi002.stdout:iscsi.foo.smithi002.vmouem smithi002 running (94s) 63s ago 11m 48.3M - 3.5 e1d6a67b021e e7f7bcd32d4e 2024-09-17T00:30:11.182 INFO:teuthology.orchestra.run.smithi002.stdout:mgr.x smithi099 *:8443,9283,8765 running (6m) 30s ago 17m 443M - 19.1.1-330-gc2ddf972 ebb0615149de 6aa2ccef9ca0 2024-09-17T00:30:11.182 INFO:teuthology.orchestra.run.smithi002.stdout:mgr.y smithi002 *:8443,9283,8765 running (7m) 63s ago 19m 509M - 19.1.1-330-gc2ddf972 ebb0615149de 8838897e8449 2024-09-17T00:30:11.183 INFO:teuthology.orchestra.run.smithi002.stdout:mon.a smithi002 running (6m) 63s ago 19m 56.9M 2048M 19.1.1-330-gc2ddf972 ebb0615149de a035def53ef6 2024-09-17T00:30:11.183 INFO:teuthology.orchestra.run.smithi002.stdout:mon.b smithi099 running (5m) 30s ago 17m 43.1M 2048M 19.1.1-330-gc2ddf972 ebb0615149de aab3ab73a850 2024-09-17T00:30:11.183 INFO:teuthology.orchestra.run.smithi002.stdout:mon.c smithi002 running (6m) 63s ago 17m 46.9M 2048M 19.1.1-330-gc2ddf972 ebb0615149de 8a510255ae66 2024-09-17T00:30:11.183 INFO:teuthology.orchestra.run.smithi002.stdout:node-exporter.a smithi002 *:9100 running (8m) 63s ago 13m 15.4M - 1.5.0 0da6a335fe13 1030ae01bb32 2024-09-17T00:30:11.183 INFO:teuthology.orchestra.run.smithi002.stdout:node-exporter.b smithi099 *:9100 running (7m) 30s ago 13m 16.9M - 1.5.0 0da6a335fe13 01fcd35dca8a 2024-09-17T00:30:11.183 INFO:teuthology.orchestra.run.smithi002.stdout:osd.0 smithi002 running (5m) 63s ago 17m 51.4M 2737M 19.1.1-330-gc2ddf972 ebb0615149de 8345719cb6f0 2024-09-17T00:30:11.183 INFO:teuthology.orchestra.run.smithi002.stdout:osd.1 smithi002 running (4m) 63s ago 16m 57.7M 2737M 19.1.1-330-gc2ddf972 ebb0615149de 3dbc23d57c53 2024-09-17T00:30:11.183 INFO:teuthology.orchestra.run.smithi002.stdout:osd.2 smithi002 running (3m) 63s ago 16m 53.1M 2737M 19.1.1-330-gc2ddf972 ebb0615149de d21401b6763b 2024-09-17T00:30:11.183 INFO:teuthology.orchestra.run.smithi002.stdout:osd.3 smithi002 running (3m) 63s ago 15m 51.4M 2737M 19.1.1-330-gc2ddf972 ebb0615149de 980f2a25d096 2024-09-17T00:30:11.183 INFO:teuthology.orchestra.run.smithi002.stdout:osd.4 smithi099 running (2m) 30s ago 15m 50.7M 2992M 19.1.1-330-gc2ddf972 ebb0615149de f644e5be4db8 2024-09-17T00:30:11.183 INFO:teuthology.orchestra.run.smithi002.stdout:osd.5 smithi099 running (107s) 30s ago 15m 52.0M 2992M 19.1.1-330-gc2ddf972 ebb0615149de 2df0d88d16ce 2024-09-17T00:30:11.183 INFO:teuthology.orchestra.run.smithi002.stdout:osd.6 smithi099 running (68s) 30s ago 14m 44.2M 2992M 19.1.1-330-gc2ddf972 ebb0615149de 7462bdbdf29b 2024-09-17T00:30:11.183 INFO:teuthology.orchestra.run.smithi002.stdout:osd.7 smithi099 running (34s) 30s ago 14m 14.5M 2992M 19.1.1-330-gc2ddf972 ebb0615149de 9821b8012a7e 2024-09-17T00:30:11.183 INFO:teuthology.orchestra.run.smithi002.stdout:prometheus.a smithi099 *:9095 running (6m) 30s ago 13m 67.3M - 2.43.0 a07b618ecd1d bf32c2ae11dc 2024-09-17T00:30:11.183 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.foo.smithi002.uwdhcs smithi002 *:8000 starting - - - - 2024-09-17T00:30:11.184 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.foo.smithi099.utzfap smithi099 *:8000 starting - - - - 2024-09-17T00:30:11.184 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.smpl.smithi002.rqnxii smithi002 *:80 starting - - - - 2024-09-17T00:30:11.184 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.smpl.smithi099.nzvkpm smithi099 *:80 starting - - - - 2024-09-17T00:30:11.404 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:30:11 smithi002 ceph-mon[63804]: pgmap v247: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 163 KiB/s rd, 170 B/s wr, 257 op/s 2024-09-17T00:30:11.404 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:30:11 smithi002 ceph-mon[63804]: from='client.34965 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:30:11.405 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:30:11 smithi002 ceph-mon[66732]: pgmap v247: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 163 KiB/s rd, 170 B/s wr, 257 op/s 2024-09-17T00:30:11.405 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:30:11 smithi002 ceph-mon[66732]: from='client.34965 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:30:11.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:30:11 smithi099 ceph-mon[56712]: pgmap v247: 161 pgs: 161 active+clean; 585 KiB data, 282 MiB used, 715 GiB / 715 GiB avail; 163 KiB/s rd, 170 B/s wr, 257 op/s 2024-09-17T00:30:11.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:30:11 smithi099 ceph-mon[56712]: from='client.34965 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:30:11.615 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-09-17T00:30:11.615 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": { 2024-09-17T00:30:11.616 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 19.1.1-330-gc2ddf972 (c2ddf9721ef254645201cdbabd9c8f49c30a586f) squid (stable)": 3 2024-09-17T00:30:11.616 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-09-17T00:30:11.616 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": { 2024-09-17T00:30:11.616 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 19.1.1-330-gc2ddf972 (c2ddf9721ef254645201cdbabd9c8f49c30a586f) squid (stable)": 2 2024-09-17T00:30:11.616 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-09-17T00:30:11.616 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": { 2024-09-17T00:30:11.616 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 19.1.1-330-gc2ddf972 (c2ddf9721ef254645201cdbabd9c8f49c30a586f) squid (stable)": 8 2024-09-17T00:30:11.616 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-09-17T00:30:11.616 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw": { 2024-09-17T00:30:11.616 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 19.1.1-330-gc2ddf972 (c2ddf9721ef254645201cdbabd9c8f49c30a586f) squid (stable)": 4 2024-09-17T00:30:11.616 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-09-17T00:30:11.616 INFO:teuthology.orchestra.run.smithi002.stdout: "overall": { 2024-09-17T00:30:11.616 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 19.1.1-330-gc2ddf972 (c2ddf9721ef254645201cdbabd9c8f49c30a586f) squid (stable)": 17 2024-09-17T00:30:11.617 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-09-17T00:30:11.617 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-09-17T00:30:11.988 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-09-17T00:30:11.988 INFO:teuthology.orchestra.run.smithi002.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f", 2024-09-17T00:30:11.988 INFO:teuthology.orchestra.run.smithi002.stdout: "in_progress": true, 2024-09-17T00:30:11.989 INFO:teuthology.orchestra.run.smithi002.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-17T00:30:11.989 INFO:teuthology.orchestra.run.smithi002.stdout: "services_complete": [ 2024-09-17T00:30:11.989 INFO:teuthology.orchestra.run.smithi002.stdout: "osd", 2024-09-17T00:30:11.989 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr", 2024-09-17T00:30:11.989 INFO:teuthology.orchestra.run.smithi002.stdout: "mon" 2024-09-17T00:30:11.989 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-09-17T00:30:11.989 INFO:teuthology.orchestra.run.smithi002.stdout: "progress": "15/23 daemons upgraded", 2024-09-17T00:30:11.989 INFO:teuthology.orchestra.run.smithi002.stdout: "message": "Currently upgrading rgw daemons", 2024-09-17T00:30:11.989 INFO:teuthology.orchestra.run.smithi002.stdout: "is_paused": false 2024-09-17T00:30:11.989 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-09-17T00:30:12.412 INFO:teuthology.orchestra.run.smithi002.stdout:HEALTH_OK 2024-09-17T00:30:12.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:30:12 smithi002 ceph-mon[63804]: from='client.34971 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:30:12.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:30:12 smithi002 ceph-mon[63804]: from='client.44869 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:30:12.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:30:12 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:30:12.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:30:12 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:30:12.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:30:12 smithi002 ceph-mon[63804]: from='client.? 172.21.15.2:0/697554772' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:30:12.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:30:12 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T00:30:12.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:30:12 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:30:12.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:30:12 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:30:12.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:30:12 smithi002 ceph-mon[63804]: from='client.? 172.21.15.2:0/1026574153' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T00:30:12.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:30:12 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:30:12.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:30:12 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:30:12.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:30:12 smithi002 ceph-mon[66732]: from='client.34971 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:30:12.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:30:12 smithi002 ceph-mon[66732]: from='client.44869 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:30:12.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:30:12 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:30:12.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:30:12 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:30:12.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:30:12 smithi002 ceph-mon[66732]: from='client.? 172.21.15.2:0/697554772' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:30:12.731 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:30:12 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T00:30:12.731 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:30:12 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:30:12.731 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:30:12 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:30:12.731 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:30:12 smithi002 ceph-mon[66732]: from='client.? 172.21.15.2:0/1026574153' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T00:30:12.731 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:30:12 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:30:12.731 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:30:12 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:30:12.968 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:30:12 smithi099 ceph-mon[56712]: from='client.34971 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:30:12.968 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:30:12 smithi099 ceph-mon[56712]: from='client.44869 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:30:12.968 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:30:12 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:30:12.968 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:30:12 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:30:12.968 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:30:12 smithi099 ceph-mon[56712]: from='client.? 172.21.15.2:0/697554772' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:30:12.968 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:30:12 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T00:30:12.968 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:30:12 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:30:12.968 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:30:12 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:30:12.968 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:30:12 smithi099 ceph-mon[56712]: from='client.? 172.21.15.2:0/1026574153' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T00:30:12.968 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:30:12 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:30:12.968 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:30:12 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:30:13.960 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:30:13 smithi099 ceph-mon[56712]: from='client.34989 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:30:13.960 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:30:13 smithi099 ceph-mon[56712]: pgmap v248: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 192 KiB/s rd, 255 B/s wr, 303 op/s 2024-09-17T00:30:13.961 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:30:13 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:30:13.961 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:30:13 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:30:13.971 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:30:13 smithi002 ceph-mon[66732]: from='client.34989 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:30:13.971 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:30:13 smithi002 ceph-mon[66732]: pgmap v248: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 192 KiB/s rd, 255 B/s wr, 303 op/s 2024-09-17T00:30:13.972 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:30:13 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:30:13.972 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:30:13 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:30:13.972 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:30:13 smithi002 ceph-mon[63804]: from='client.34989 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:30:13.972 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:30:13 smithi002 ceph-mon[63804]: pgmap v248: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 192 KiB/s rd, 255 B/s wr, 303 op/s 2024-09-17T00:30:13.972 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:30:13 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:30:13.972 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:30:13 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:30:15.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:30:15 smithi099 ceph-mon[56712]: pgmap v249: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 157 KiB/s rd, 170 B/s wr, 248 op/s 2024-09-17T00:30:15.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:30:15 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:30:15.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:30:15 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:30:15.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:30:15 smithi002 ceph-mon[63804]: pgmap v249: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 157 KiB/s rd, 170 B/s wr, 248 op/s 2024-09-17T00:30:15.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:30:15 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:30:15.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:30:15 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:30:15.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:30:15 smithi002 ceph-mon[66732]: pgmap v249: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 157 KiB/s rd, 170 B/s wr, 248 op/s 2024-09-17T00:30:15.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:30:15 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:30:15.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:30:15 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:30:16.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:30:16 smithi099 ceph-mon[56712]: Detected new or changed devices on smithi099 2024-09-17T00:30:16.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:30:16 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:30:16.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:30:16 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:30:16.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:30:16 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:30:16.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:30:16 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:30:16.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:30:16 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:30:16.979 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:30:16 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: ::ffff:172.21.15.99 - - [17/Sep/2024:00:30:16] "GET /metrics HTTP/1.1" 200 38591 "" "Prometheus/2.43.0" 2024-09-17T00:30:16.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:30:16 smithi002 ceph-mon[66732]: Detected new or changed devices on smithi099 2024-09-17T00:30:16.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:30:16 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:30:16.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:30:16 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:30:16.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:30:16 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:30:16.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:30:16 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:30:16.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:30:16 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:30:16.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:30:16 smithi002 ceph-mon[63804]: Detected new or changed devices on smithi099 2024-09-17T00:30:16.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:30:16 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:30:16.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:30:16 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:30:16.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:30:16 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:30:16.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:30:16 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:30:16.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:30:16 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:30:17.622 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:30:17 smithi002 ceph-mon[63804]: pgmap v250: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 129 KiB/s rd, 170 B/s wr, 203 op/s 2024-09-17T00:30:17.622 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:30:17 smithi002 ceph-mon[63804]: Detected new or changed devices on smithi002 2024-09-17T00:30:17.622 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:30:17 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T00:30:17.622 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:30:17 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:30:17.622 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:30:17 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:30:17.622 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:30:17 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:30:17.622 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:30:17 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:30:17.622 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:30:17 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:30:17.622 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:30:17 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:30:17.622 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:30:17 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:30:17.623 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:30:17 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi002.uwdhcs"}]: dispatch 2024-09-17T00:30:17.623 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:30:17 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi002.uwdhcs"}]: dispatch 2024-09-17T00:30:17.623 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:30:17 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi002.uwdhcs"}]': finished 2024-09-17T00:30:17.623 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:30:17 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi099.utzfap"}]: dispatch 2024-09-17T00:30:17.623 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:30:17 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi099.utzfap"}]: dispatch 2024-09-17T00:30:17.623 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:30:17 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi099.utzfap"}]': finished 2024-09-17T00:30:17.623 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:30:17 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi002.rqnxii"}]: dispatch 2024-09-17T00:30:17.623 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:30:17 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi002.rqnxii"}]: dispatch 2024-09-17T00:30:17.623 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:30:17 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi002.rqnxii"}]': finished 2024-09-17T00:30:17.623 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:30:17 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi099.nzvkpm"}]: dispatch 2024-09-17T00:30:17.623 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:30:17 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi099.nzvkpm"}]: dispatch 2024-09-17T00:30:17.623 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:30:17 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi099.nzvkpm"}]': finished 2024-09-17T00:30:17.623 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:30:17 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:30:17.623 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:30:17 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:30:17.623 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:30:17 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:30:17.624 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:30:17 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:30:17.624 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:30:17 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:30:17.624 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:30:17 smithi002 ceph-mon[66732]: pgmap v250: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 129 KiB/s rd, 170 B/s wr, 203 op/s 2024-09-17T00:30:17.624 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:30:17 smithi002 ceph-mon[66732]: Detected new or changed devices on smithi002 2024-09-17T00:30:17.624 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:30:17 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T00:30:17.624 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:30:17 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:30:17.624 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:30:17 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:30:17.624 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:30:17 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:30:17.624 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:30:17 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:30:17.625 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:30:17 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:30:17.625 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:30:17 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:30:17.625 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:30:17 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:30:17.625 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:30:17 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi002.uwdhcs"}]: dispatch 2024-09-17T00:30:17.625 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:30:17 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi002.uwdhcs"}]: dispatch 2024-09-17T00:30:17.625 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:30:17 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi002.uwdhcs"}]': finished 2024-09-17T00:30:17.625 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:30:17 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi099.utzfap"}]: dispatch 2024-09-17T00:30:17.625 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:30:17 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi099.utzfap"}]: dispatch 2024-09-17T00:30:17.625 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:30:17 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi099.utzfap"}]': finished 2024-09-17T00:30:17.625 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:30:17 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi002.rqnxii"}]: dispatch 2024-09-17T00:30:17.625 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:30:17 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi002.rqnxii"}]: dispatch 2024-09-17T00:30:17.625 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:30:17 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi002.rqnxii"}]': finished 2024-09-17T00:30:17.625 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:30:17 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi099.nzvkpm"}]: dispatch 2024-09-17T00:30:17.625 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:30:17 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi099.nzvkpm"}]: dispatch 2024-09-17T00:30:17.626 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:30:17 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi099.nzvkpm"}]': finished 2024-09-17T00:30:17.626 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:30:17 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:30:17.626 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:30:17 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:30:17.626 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:30:17 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:30:17.626 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:30:17 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:30:17.626 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:30:17 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:30:17.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:30:17 smithi099 ceph-mon[56712]: pgmap v250: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 129 KiB/s rd, 170 B/s wr, 203 op/s 2024-09-17T00:30:17.968 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:30:17 smithi099 ceph-mon[56712]: Detected new or changed devices on smithi002 2024-09-17T00:30:17.968 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:30:17 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-17T00:30:17.968 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:30:17 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:30:17.968 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:30:17 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:30:17.968 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:30:17 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:30:17.968 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:30:17 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:30:17.968 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:30:17 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:30:17.968 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:30:17 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:30:17.969 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:30:17 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:30:17.969 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:30:17 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi002.uwdhcs"}]: dispatch 2024-09-17T00:30:17.969 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:30:17 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi002.uwdhcs"}]: dispatch 2024-09-17T00:30:17.969 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:30:17 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi002.uwdhcs"}]': finished 2024-09-17T00:30:17.969 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:30:17 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi099.utzfap"}]: dispatch 2024-09-17T00:30:17.969 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:30:17 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi099.utzfap"}]: dispatch 2024-09-17T00:30:17.969 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:30:17 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi099.utzfap"}]': finished 2024-09-17T00:30:17.969 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:30:17 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi002.rqnxii"}]: dispatch 2024-09-17T00:30:17.969 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:30:17 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi002.rqnxii"}]: dispatch 2024-09-17T00:30:17.970 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:30:17 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi002.rqnxii"}]': finished 2024-09-17T00:30:17.970 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:30:17 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi099.nzvkpm"}]: dispatch 2024-09-17T00:30:17.970 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:30:17 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi099.nzvkpm"}]: dispatch 2024-09-17T00:30:17.970 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:30:17 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi099.nzvkpm"}]': finished 2024-09-17T00:30:17.970 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:30:17 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:30:17.970 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:30:17 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:30:17.970 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:30:17 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:30:17.971 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:30:17 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:30:17.971 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:30:17 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:30:18.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:30:18 smithi099 ceph-mon[56712]: Upgrade: Setting container_image for all rgw 2024-09-17T00:30:18.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:30:18 smithi099 ceph-mon[56712]: Upgrade: Setting container_image for all rbd-mirror 2024-09-17T00:30:18.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:30:18 smithi099 ceph-mon[56712]: Upgrade: Setting container_image for all ceph-exporter 2024-09-17T00:30:18.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:30:18 smithi099 ceph-mon[56712]: from='client.54506 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:30:18.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:30:18 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:30:18.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:30:18 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi002.vmouem", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-17T00:30:18.968 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:30:18 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi002.vmouem", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-17T00:30:18.968 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:30:18 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:30:18.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:30:18 smithi002 ceph-mon[63804]: Upgrade: Setting container_image for all rgw 2024-09-17T00:30:18.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:30:18 smithi002 ceph-mon[63804]: Upgrade: Setting container_image for all rbd-mirror 2024-09-17T00:30:18.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:30:18 smithi002 ceph-mon[63804]: Upgrade: Setting container_image for all ceph-exporter 2024-09-17T00:30:18.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:30:18 smithi002 ceph-mon[63804]: from='client.54506 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:30:18.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:30:18 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:30:18.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:30:18 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi002.vmouem", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-17T00:30:18.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:30:18 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi002.vmouem", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-17T00:30:18.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:30:18 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:30:18.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:30:18 smithi002 ceph-mon[66732]: Upgrade: Setting container_image for all rgw 2024-09-17T00:30:18.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:30:18 smithi002 ceph-mon[66732]: Upgrade: Setting container_image for all rbd-mirror 2024-09-17T00:30:18.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:30:18 smithi002 ceph-mon[66732]: Upgrade: Setting container_image for all ceph-exporter 2024-09-17T00:30:18.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:30:18 smithi002 ceph-mon[66732]: from='client.54506 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:30:18.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:30:18 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:30:18.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:30:18 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi002.vmouem", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-17T00:30:18.981 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:30:18 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi002.vmouem", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-17T00:30:18.981 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:30:18 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:30:19.873 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:30:19 smithi002 ceph-mon[66732]: Upgrade: Updating iscsi.foo.smithi002.vmouem 2024-09-17T00:30:19.873 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:30:19 smithi002 ceph-mon[66732]: Deploying daemon iscsi.foo.smithi002.vmouem on smithi002 2024-09-17T00:30:19.873 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:30:19 smithi002 ceph-mon[66732]: pgmap v251: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 118 KiB/s rd, 170 B/s wr, 186 op/s 2024-09-17T00:30:19.873 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:30:19 smithi002 ceph-mon[63804]: Upgrade: Updating iscsi.foo.smithi002.vmouem 2024-09-17T00:30:19.873 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:30:19 smithi002 ceph-mon[63804]: Deploying daemon iscsi.foo.smithi002.vmouem on smithi002 2024-09-17T00:30:19.874 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:30:19 smithi002 ceph-mon[63804]: pgmap v251: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 118 KiB/s rd, 170 B/s wr, 186 op/s 2024-09-17T00:30:19.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:30:19 smithi099 ceph-mon[56712]: Upgrade: Updating iscsi.foo.smithi002.vmouem 2024-09-17T00:30:19.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:30:19 smithi099 ceph-mon[56712]: Deploying daemon iscsi.foo.smithi002.vmouem on smithi002 2024-09-17T00:30:19.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:30:19 smithi099 ceph-mon[56712]: pgmap v251: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 118 KiB/s rd, 170 B/s wr, 186 op/s 2024-09-17T00:30:21.813 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:30:21 smithi002 ceph-mon[63804]: pgmap v252: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 79 KiB/s rd, 85 B/s wr, 124 op/s 2024-09-17T00:30:21.814 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:30:21 smithi002 ceph-mon[66732]: pgmap v252: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 79 KiB/s rd, 85 B/s wr, 124 op/s 2024-09-17T00:30:21.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:30:21 smithi099 ceph-mon[56712]: pgmap v252: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 79 KiB/s rd, 85 B/s wr, 124 op/s 2024-09-17T00:30:22.229 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:30:21 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: 2024-09-17T00:30:21.858+0000 7f17e1e2b640 -1 log_channel(cephadm) log [ERR] : Upgrade: Paused due to UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi002.vmouem on host smithi002 failed. 2024-09-17T00:30:22.935 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:30:22 smithi002 ceph-mon[66732]: Upgrade: Paused due to UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi002.vmouem on host smithi002 failed. 2024-09-17T00:30:22.935 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:30:22 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:30:22.935 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:30:22 smithi002 ceph-mon[66732]: pgmap v253: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 63 KiB/s rd, 88 B/s wr, 98 op/s 2024-09-17T00:30:22.953 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:30:22 smithi002 ceph-mon[63804]: Upgrade: Paused due to UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi002.vmouem on host smithi002 failed. 2024-09-17T00:30:22.953 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:30:22 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:30:22.953 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:30:22 smithi002 ceph-mon[63804]: pgmap v253: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 63 KiB/s rd, 88 B/s wr, 98 op/s 2024-09-17T00:30:23.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:30:22 smithi099 ceph-mon[56712]: Upgrade: Paused due to UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi002.vmouem on host smithi002 failed. 2024-09-17T00:30:23.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:30:22 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:30:23.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:30:22 smithi099 ceph-mon[56712]: pgmap v253: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 63 KiB/s rd, 88 B/s wr, 98 op/s 2024-09-17T00:30:24.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:30:23 smithi099 ceph-mon[56712]: Health check failed: Upgrading daemon iscsi.foo.smithi002.vmouem on host smithi002 failed. (UPGRADE_REDEPLOY_DAEMON) 2024-09-17T00:30:24.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:30:23 smithi002 ceph-mon[66732]: Health check failed: Upgrading daemon iscsi.foo.smithi002.vmouem on host smithi002 failed. (UPGRADE_REDEPLOY_DAEMON) 2024-09-17T00:30:24.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:30:23 smithi002 ceph-mon[63804]: Health check failed: Upgrading daemon iscsi.foo.smithi002.vmouem on host smithi002 failed. (UPGRADE_REDEPLOY_DAEMON) 2024-09-17T00:30:24.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:30:24 smithi002 ceph-mon[66732]: pgmap v254: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 0 B/s wr, 21 op/s 2024-09-17T00:30:24.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:30:24 smithi002 ceph-mon[63804]: pgmap v254: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 0 B/s wr, 21 op/s 2024-09-17T00:30:25.217 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:30:24 smithi099 ceph-mon[56712]: pgmap v254: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 0 B/s wr, 21 op/s 2024-09-17T00:30:26.642 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:30:26 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:30:26.642 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:30:26 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:30:26.642 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:30:26 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:30:26.642 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:30:26 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:30:26.643 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:30:26 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:30:26.643 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:30:26 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:30:26.643 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:30:26 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:30:26.643 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:30:26 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:30:26.643 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:30:26 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:30:26.643 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:30:26 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:30:26.643 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:30:26 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:30:26.643 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:30:26 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:30:26.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:30:26 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:30:26.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:30:26 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:30:26.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:30:26 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:30:26.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:30:26 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:30:26.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:30:26 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:30:26.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:30:26 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:30:26.979 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:30:26 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: ::ffff:172.21.15.99 - - [17/Sep/2024:00:30:26] "GET /metrics HTTP/1.1" 200 38591 "" "Prometheus/2.43.0" 2024-09-17T00:30:27.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:30:27 smithi099 ceph-mon[56712]: pgmap v255: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 704 B/s rd, 0 op/s 2024-09-17T00:30:27.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:30:27 smithi099 ceph-mon[56712]: pgmap v256: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 832 B/s rd, 0 op/s 2024-09-17T00:30:27.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:30:27 smithi099 ceph-mon[56712]: Checking dashboard <-> RGW credentials 2024-09-17T00:30:27.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:30:27 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:30:27.717 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:30:27 smithi099 ceph-mon[56712]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-09-17T00:30:27.718 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:30:27 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:30:27.718 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:30:27 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:30:27.718 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:30:27 smithi099 ceph-mon[56712]: pgmap v257: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 1000 B/s rd, 1 op/s 2024-09-17T00:30:27.718 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:30:27 smithi099 ceph-mon[56712]: pgmap v258: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 496 B/s rd, 0 op/s 2024-09-17T00:30:27.718 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:30:27 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:30:27.718 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:30:27 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:30:27.718 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:30:27 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T00:30:27.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:30:27 smithi002 ceph-mon[66732]: pgmap v255: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 704 B/s rd, 0 op/s 2024-09-17T00:30:27.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:30:27 smithi002 ceph-mon[66732]: pgmap v256: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 832 B/s rd, 0 op/s 2024-09-17T00:30:27.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:30:27 smithi002 ceph-mon[66732]: Checking dashboard <-> RGW credentials 2024-09-17T00:30:27.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:30:27 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:30:27.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:30:27 smithi002 ceph-mon[66732]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-09-17T00:30:27.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:30:27 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:30:27.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:30:27 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:30:27.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:30:27 smithi002 ceph-mon[66732]: pgmap v257: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 1000 B/s rd, 1 op/s 2024-09-17T00:30:27.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:30:27 smithi002 ceph-mon[66732]: pgmap v258: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 496 B/s rd, 0 op/s 2024-09-17T00:30:27.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:30:27 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:30:27.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:30:27 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:30:27.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:30:27 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T00:30:27.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:30:27 smithi002 ceph-mon[63804]: pgmap v255: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 704 B/s rd, 0 op/s 2024-09-17T00:30:27.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:30:27 smithi002 ceph-mon[63804]: pgmap v256: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 832 B/s rd, 0 op/s 2024-09-17T00:30:27.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:30:27 smithi002 ceph-mon[63804]: Checking dashboard <-> RGW credentials 2024-09-17T00:30:27.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:30:27 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:30:27.731 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:30:27 smithi002 ceph-mon[63804]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-09-17T00:30:27.731 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:30:27 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:30:27.731 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:30:27 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:30:27.731 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:30:27 smithi002 ceph-mon[63804]: pgmap v257: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 1000 B/s rd, 1 op/s 2024-09-17T00:30:27.731 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:30:27 smithi002 ceph-mon[63804]: pgmap v258: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 496 B/s rd, 0 op/s 2024-09-17T00:30:27.731 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:30:27 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:30:27.731 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:30:27 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:30:27.731 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:30:27 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T00:30:29.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:30:29 smithi002 ceph-mon[63804]: pgmap v259: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 3 op/s 2024-09-17T00:30:29.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:30:29 smithi002 ceph-mon[66732]: pgmap v259: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 3 op/s 2024-09-17T00:30:29.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:30:29 smithi099 ceph-mon[56712]: pgmap v259: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 3 op/s 2024-09-17T00:30:31.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:30:31 smithi002 ceph-mon[66732]: pgmap v260: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 2.6 KiB/s rd, 2 op/s 2024-09-17T00:30:31.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:30:31 smithi002 ceph-mon[63804]: pgmap v260: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 2.6 KiB/s rd, 2 op/s 2024-09-17T00:30:31.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:30:31 smithi099 ceph-mon[56712]: pgmap v260: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 2.6 KiB/s rd, 2 op/s 2024-09-17T00:30:33.905 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:30:33 smithi002 ceph-mon[66732]: pgmap v261: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 3 op/s 2024-09-17T00:30:33.905 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:30:33 smithi002 ceph-mon[66732]: from='client.? 172.21.15.2:0/3929085289' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-17T00:30:33.905 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:30:33 smithi002 ceph-mon[63804]: pgmap v261: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 3 op/s 2024-09-17T00:30:33.905 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:30:33 smithi002 ceph-mon[63804]: from='client.? 172.21.15.2:0/3929085289' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-17T00:30:33.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:30:33 smithi099 ceph-mon[56712]: pgmap v261: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 3.4 KiB/s rd, 3 op/s 2024-09-17T00:30:33.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:30:33 smithi099 ceph-mon[56712]: from='client.? 172.21.15.2:0/3929085289' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-17T00:30:35.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:30:35 smithi099 ceph-mon[56712]: pgmap v262: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 2.8 KiB/s rd, 3 op/s 2024-09-17T00:30:35.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:30:35 smithi002 ceph-mon[63804]: pgmap v262: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 2.8 KiB/s rd, 3 op/s 2024-09-17T00:30:35.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:30:35 smithi002 ceph-mon[66732]: pgmap v262: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 2.8 KiB/s rd, 3 op/s 2024-09-17T00:30:36.897 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:30:36 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: ::ffff:172.21.15.99 - - [17/Sep/2024:00:30:36] "GET /metrics HTTP/1.1" 200 38705 "" "Prometheus/2.43.0" 2024-09-17T00:30:37.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:30:37 smithi002 ceph-mon[66732]: pgmap v263: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 2.3 KiB/s rd, 2 op/s 2024-09-17T00:30:37.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:30:37 smithi002 ceph-mon[63804]: pgmap v263: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 2.3 KiB/s rd, 2 op/s 2024-09-17T00:30:37.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:30:37 smithi099 ceph-mon[56712]: pgmap v263: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 2.3 KiB/s rd, 2 op/s 2024-09-17T00:30:39.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:30:39 smithi099 ceph-mon[56712]: pgmap v264: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 2 op/s 2024-09-17T00:30:39.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:30:39 smithi002 ceph-mon[66732]: pgmap v264: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 2 op/s 2024-09-17T00:30:39.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:30:39 smithi002 ceph-mon[63804]: pgmap v264: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 2 op/s 2024-09-17T00:30:41.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:30:41 smithi099 ceph-mon[56712]: pgmap v265: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-17T00:30:41.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:30:41 smithi002 ceph-mon[63804]: pgmap v265: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-17T00:30:41.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:30:41 smithi002 ceph-mon[66732]: pgmap v265: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-17T00:30:42.800 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:30:42 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T00:30:42.801 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:30:42 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T00:30:42.801 INFO:teuthology.orchestra.run.smithi002.stdout:true 2024-09-17T00:30:42.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:30:42 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T00:30:43.149 INFO:teuthology.orchestra.run.smithi002.stdout:"Error: UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi002.vmouem on host smithi002 failed." 2024-09-17T00:30:43.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:30:43 smithi002 ceph-mon[63804]: pgmap v266: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 1 op/s 2024-09-17T00:30:43.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:30:43 smithi002 ceph-mon[66732]: pgmap v266: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 1 op/s 2024-09-17T00:30:43.900 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3aad27a2-7489-11ef-bceb-c7b262605968 -e sha1=c2ddf9721ef254645201cdbabd9c8f49c30a586f -- bash -c 'ceph orch ps' 2024-09-17T00:30:43.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:30:43 smithi099 ceph-mon[56712]: pgmap v266: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 1 op/s 2024-09-17T00:30:44.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:30:44 smithi002 ceph-mon[66732]: from='client.44953 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:30:44.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:30:44 smithi002 ceph-mon[66732]: from='client.44938 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:30:44.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:30:44 smithi002 ceph-mon[66732]: from='client.44959 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:30:44.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:30:44 smithi002 ceph-mon[63804]: from='client.44953 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:30:44.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:30:44 smithi002 ceph-mon[63804]: from='client.44938 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:30:44.730 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:30:44 smithi002 ceph-mon[63804]: from='client.44959 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:30:44.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:30:44 smithi099 ceph-mon[56712]: from='client.44953 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:30:44.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:30:44 smithi099 ceph-mon[56712]: from='client.44938 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:30:44.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:30:44 smithi099 ceph-mon[56712]: from='client.44959 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:30:45.578 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:30:45 smithi002 ceph-mon[63804]: pgmap v267: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-09-17T00:30:45.578 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:30:45 smithi002 ceph-mon[66732]: pgmap v267: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-09-17T00:30:45.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:30:45 smithi099 ceph-mon[56712]: pgmap v267: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-09-17T00:30:46.052 INFO:teuthology.orchestra.run.smithi002.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-17T00:30:46.052 INFO:teuthology.orchestra.run.smithi002.stdout:alertmanager.a smithi002 *:9093,9094 running (8m) 19s ago 13m - - 0.25.0 c8568f914cd2 a21e13b3d677 2024-09-17T00:30:46.052 INFO:teuthology.orchestra.run.smithi002.stdout:grafana.a smithi099 *:3000 running (8m) 20s ago 13m 54.4M - 8.3.5 dad864ee21e9 bafb824a3c08 2024-09-17T00:30:46.052 INFO:teuthology.orchestra.run.smithi002.stdout:iscsi.foo.smithi002.vmouem smithi002 unknown 19s ago 12m - - 2024-09-17T00:30:46.052 INFO:teuthology.orchestra.run.smithi002.stdout:mgr.x smithi099 *:8443,9283,8765 running (7m) 20s ago 18m 443M - 19.1.1-330-gc2ddf972 ebb0615149de 6aa2ccef9ca0 2024-09-17T00:30:46.052 INFO:teuthology.orchestra.run.smithi002.stdout:mgr.y smithi002 *:8443,9283,8765 running (8m) 19s ago 19m - - 19.1.1-330-gc2ddf972 ebb0615149de 8838897e8449 2024-09-17T00:30:46.052 INFO:teuthology.orchestra.run.smithi002.stdout:mon.a smithi002 running (7m) 19s ago 19m - 2048M 19.1.1-330-gc2ddf972 ebb0615149de a035def53ef6 2024-09-17T00:30:46.052 INFO:teuthology.orchestra.run.smithi002.stdout:mon.b smithi099 running (6m) 20s ago 18m 46.0M 2048M 19.1.1-330-gc2ddf972 ebb0615149de aab3ab73a850 2024-09-17T00:30:46.052 INFO:teuthology.orchestra.run.smithi002.stdout:mon.c smithi002 running (6m) 19s ago 18m - 2048M 19.1.1-330-gc2ddf972 ebb0615149de 8a510255ae66 2024-09-17T00:30:46.053 INFO:teuthology.orchestra.run.smithi002.stdout:node-exporter.a smithi002 *:9100 running (8m) 19s ago 14m - - 1.5.0 0da6a335fe13 1030ae01bb32 2024-09-17T00:30:46.053 INFO:teuthology.orchestra.run.smithi002.stdout:node-exporter.b smithi099 *:9100 running (8m) 20s ago 14m 16.5M - 1.5.0 0da6a335fe13 01fcd35dca8a 2024-09-17T00:30:46.053 INFO:teuthology.orchestra.run.smithi002.stdout:osd.0 smithi002 running (5m) 19s ago 17m - 2737M 19.1.1-330-gc2ddf972 ebb0615149de 8345719cb6f0 2024-09-17T00:30:46.053 INFO:teuthology.orchestra.run.smithi002.stdout:osd.1 smithi002 running (4m) 19s ago 17m - 2737M 19.1.1-330-gc2ddf972 ebb0615149de 3dbc23d57c53 2024-09-17T00:30:46.053 INFO:teuthology.orchestra.run.smithi002.stdout:osd.2 smithi002 running (4m) 19s ago 16m - 2737M 19.1.1-330-gc2ddf972 ebb0615149de d21401b6763b 2024-09-17T00:30:46.053 INFO:teuthology.orchestra.run.smithi002.stdout:osd.3 smithi002 running (3m) 19s ago 16m - 2737M 19.1.1-330-gc2ddf972 ebb0615149de 980f2a25d096 2024-09-17T00:30:46.053 INFO:teuthology.orchestra.run.smithi002.stdout:osd.4 smithi099 running (2m) 20s ago 16m 59.0M 2992M 19.1.1-330-gc2ddf972 ebb0615149de f644e5be4db8 2024-09-17T00:30:46.053 INFO:teuthology.orchestra.run.smithi002.stdout:osd.5 smithi099 running (2m) 20s ago 15m 57.5M 2992M 19.1.1-330-gc2ddf972 ebb0615149de 2df0d88d16ce 2024-09-17T00:30:46.053 INFO:teuthology.orchestra.run.smithi002.stdout:osd.6 smithi099 running (103s) 20s ago 15m 50.4M 2992M 19.1.1-330-gc2ddf972 ebb0615149de 7462bdbdf29b 2024-09-17T00:30:46.053 INFO:teuthology.orchestra.run.smithi002.stdout:osd.7 smithi099 running (69s) 20s ago 15m 62.1M 2992M 19.1.1-330-gc2ddf972 ebb0615149de 9821b8012a7e 2024-09-17T00:30:46.053 INFO:teuthology.orchestra.run.smithi002.stdout:prometheus.a smithi099 *:9095 running (7m) 20s ago 14m 67.4M - 2.43.0 a07b618ecd1d bf32c2ae11dc 2024-09-17T00:30:46.053 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.foo.smithi002.uwdhcs smithi002 *:8000 running (54s) 19s ago 12m - - 19.1.1-330-gc2ddf972 ebb0615149de 19fa1132d770 2024-09-17T00:30:46.053 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.foo.smithi099.utzfap smithi099 *:8000 running (44s) 20s ago 12m 76.4M - 19.1.1-330-gc2ddf972 ebb0615149de b56c819bfbb8 2024-09-17T00:30:46.053 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.smpl.smithi002.rqnxii smithi002 *:80 running (49s) 19s ago 12m - - 19.1.1-330-gc2ddf972 ebb0615149de 50f19f6bf716 2024-09-17T00:30:46.054 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.smpl.smithi099.nzvkpm smithi099 *:80 running (38s) 20s ago 12m 77.0M - 19.1.1-330-gc2ddf972 ebb0615149de 4256e3820d2d 2024-09-17T00:30:46.729 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:30:46 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: ::ffff:172.21.15.99 - - [17/Sep/2024:00:30:46] "GET /metrics HTTP/1.1" 200 38705 "" "Prometheus/2.43.0" 2024-09-17T00:30:46.841 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3aad27a2-7489-11ef-bceb-c7b262605968 -e sha1=c2ddf9721ef254645201cdbabd9c8f49c30a586f -- bash -c 'ceph versions' 2024-09-17T00:30:47.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:30:47 smithi002 ceph-mon[63804]: from='client.44965 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:30:47.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:30:47 smithi002 ceph-mon[63804]: pgmap v268: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:30:47.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:30:47 smithi002 ceph-mon[66732]: from='client.44965 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:30:47.730 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:30:47 smithi002 ceph-mon[66732]: pgmap v268: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:30:47.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:30:47 smithi099 ceph-mon[56712]: from='client.44965 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:30:47.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:30:47 smithi099 ceph-mon[56712]: pgmap v268: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:30:49.158 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-09-17T00:30:49.159 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": { 2024-09-17T00:30:49.159 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 19.1.1-330-gc2ddf972 (c2ddf9721ef254645201cdbabd9c8f49c30a586f) squid (stable)": 3 2024-09-17T00:30:49.159 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-09-17T00:30:49.159 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": { 2024-09-17T00:30:49.159 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 19.1.1-330-gc2ddf972 (c2ddf9721ef254645201cdbabd9c8f49c30a586f) squid (stable)": 2 2024-09-17T00:30:49.159 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-09-17T00:30:49.159 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": { 2024-09-17T00:30:49.159 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 19.1.1-330-gc2ddf972 (c2ddf9721ef254645201cdbabd9c8f49c30a586f) squid (stable)": 8 2024-09-17T00:30:49.159 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-09-17T00:30:49.159 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw": { 2024-09-17T00:30:49.159 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 19.1.1-330-gc2ddf972 (c2ddf9721ef254645201cdbabd9c8f49c30a586f) squid (stable)": 4 2024-09-17T00:30:49.160 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-09-17T00:30:49.160 INFO:teuthology.orchestra.run.smithi002.stdout: "overall": { 2024-09-17T00:30:49.160 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 19.1.1-330-gc2ddf972 (c2ddf9721ef254645201cdbabd9c8f49c30a586f) squid (stable)": 17 2024-09-17T00:30:49.160 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-09-17T00:30:49.160 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-09-17T00:30:49.836 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:30:49 smithi002 ceph-mon[63804]: pgmap v269: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:30:49.836 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:30:49 smithi002 ceph-mon[63804]: from='client.? 172.21.15.2:0/2441892764' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:30:49.836 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:30:49 smithi002 ceph-mon[66732]: pgmap v269: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:30:49.837 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:30:49 smithi002 ceph-mon[66732]: from='client.? 172.21.15.2:0/2441892764' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:30:49.861 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3aad27a2-7489-11ef-bceb-c7b262605968 -e sha1=c2ddf9721ef254645201cdbabd9c8f49c30a586f -- bash -c 'echo "wait for servicemap items w/ changing names to refresh"' 2024-09-17T00:30:49.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:30:49 smithi099 ceph-mon[56712]: pgmap v269: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:30:49.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:30:49 smithi099 ceph-mon[56712]: from='client.? 172.21.15.2:0/2441892764' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:30:51.630 INFO:teuthology.orchestra.run.smithi002.stdout:wait for servicemap items w/ changing names to refresh 2024-09-17T00:30:51.630 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:30:51 smithi002 ceph-mon[66732]: pgmap v270: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:30:51.630 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:30:51 smithi002 ceph-mon[63804]: pgmap v270: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:30:51.968 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:30:51 smithi099 ceph-mon[56712]: pgmap v270: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:30:52.314 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3aad27a2-7489-11ef-bceb-c7b262605968 -e sha1=c2ddf9721ef254645201cdbabd9c8f49c30a586f -- bash -c 'sleep 60' 2024-09-17T00:30:53.901 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:30:53 smithi002 ceph-mon[63804]: pgmap v271: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:30:53.901 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:30:53 smithi002 ceph-mon[66732]: pgmap v271: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:30:53.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:30:53 smithi099 ceph-mon[56712]: pgmap v271: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:30:54.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:30:54 smithi099 ceph-mon[56712]: from='client.44938 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:30:54.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:30:54 smithi002 ceph-mon[63804]: from='client.44938 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:30:54.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:30:54 smithi002 ceph-mon[66732]: from='client.44938 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:30:55.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:30:55 smithi002 ceph-mon[66732]: pgmap v272: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:30:55.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:30:55 smithi002 ceph-mon[63804]: pgmap v272: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:30:55.868 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:30:55 smithi099 ceph-mon[56712]: pgmap v272: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:30:56.979 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:30:56 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: ::ffff:172.21.15.99 - - [17/Sep/2024:00:30:56] "GET /metrics HTTP/1.1" 200 38705 "" "Prometheus/2.43.0" 2024-09-17T00:30:57.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:30:57 smithi099 ceph-mon[56712]: pgmap v273: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:30:57.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:30:57 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T00:30:57.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:30:57 smithi002 ceph-mon[66732]: pgmap v273: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:30:57.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:30:57 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T00:30:57.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:30:57 smithi002 ceph-mon[63804]: pgmap v273: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:30:57.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:30:57 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T00:30:59.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:30:59 smithi099 ceph-mon[56712]: pgmap v274: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:30:59.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:30:59 smithi002 ceph-mon[66732]: pgmap v274: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:30:59.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:30:59 smithi002 ceph-mon[63804]: pgmap v274: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:31:01.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:31:01 smithi099 ceph-mon[56712]: pgmap v275: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:31:01.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:31:01 smithi002 ceph-mon[66732]: pgmap v275: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:31:01.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:31:01 smithi002 ceph-mon[63804]: pgmap v275: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:31:03.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:31:03 smithi099 ceph-mon[56712]: pgmap v276: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:31:03.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:31:03 smithi002 ceph-mon[66732]: pgmap v276: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:31:03.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:31:03 smithi002 ceph-mon[63804]: pgmap v276: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:31:04.877 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:31:04 smithi002 ceph-mon[63804]: from='client.44938 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:31:04.877 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:31:04 smithi002 ceph-mon[66732]: from='client.44938 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:31:04.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:31:04 smithi099 ceph-mon[56712]: from='client.44938 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:31:05.729 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:31:05 smithi002 ceph-mon[66732]: pgmap v277: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:31:05.729 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:31:05 smithi002 ceph-mon[63804]: pgmap v277: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:31:05.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:31:05 smithi099 ceph-mon[56712]: pgmap v277: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:31:06.979 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:31:06 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: ::ffff:172.21.15.99 - - [17/Sep/2024:00:31:06] "GET /metrics HTTP/1.1" 200 38704 "" "Prometheus/2.43.0" 2024-09-17T00:31:07.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:31:07 smithi099 ceph-mon[56712]: pgmap v278: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-17T00:31:07.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:31:07 smithi002 ceph-mon[63804]: pgmap v278: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-17T00:31:07.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:31:07 smithi002 ceph-mon[66732]: pgmap v278: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-17T00:31:09.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:31:09 smithi099 ceph-mon[56712]: pgmap v279: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-17T00:31:09.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:31:09 smithi002 ceph-mon[63804]: pgmap v279: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-17T00:31:09.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:31:09 smithi002 ceph-mon[66732]: pgmap v279: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-17T00:31:11.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:31:11 smithi099 ceph-mon[56712]: pgmap v280: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:31:12.005 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:31:11 smithi002 ceph-mon[63804]: pgmap v280: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:31:12.005 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:31:11 smithi002 ceph-mon[66732]: pgmap v280: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:31:12.848 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:31:12 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T00:31:12.849 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:31:12 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T00:31:12.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:31:12 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T00:31:13.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:31:13 smithi099 ceph-mon[56712]: pgmap v281: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:31:13.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:31:13 smithi002 ceph-mon[66732]: pgmap v281: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:31:13.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:31:13 smithi002 ceph-mon[63804]: pgmap v281: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:31:14.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:31:14 smithi099 ceph-mon[56712]: from='client.44938 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:31:14.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:31:14 smithi002 ceph-mon[66732]: from='client.44938 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:31:14.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:31:14 smithi002 ceph-mon[63804]: from='client.44938 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:31:15.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:31:15 smithi099 ceph-mon[56712]: pgmap v282: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:31:15.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:31:15 smithi002 ceph-mon[63804]: pgmap v282: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:31:15.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:31:15 smithi002 ceph-mon[66732]: pgmap v282: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:31:16.979 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:31:16 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: ::ffff:172.21.15.99 - - [17/Sep/2024:00:31:16] "GET /metrics HTTP/1.1" 200 38704 "" "Prometheus/2.43.0" 2024-09-17T00:31:17.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:31:17 smithi099 ceph-mon[56712]: pgmap v283: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:31:17.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:31:17 smithi002 ceph-mon[63804]: pgmap v283: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:31:17.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:31:17 smithi002 ceph-mon[66732]: pgmap v283: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:31:19.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:31:19 smithi099 ceph-mon[56712]: pgmap v284: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:31:19.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:31:19 smithi002 ceph-mon[63804]: pgmap v284: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:31:19.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:31:19 smithi002 ceph-mon[66732]: pgmap v284: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:31:21.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:31:21 smithi099 ceph-mon[56712]: pgmap v285: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:31:21.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:31:21 smithi002 ceph-mon[63804]: pgmap v285: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:31:21.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:31:21 smithi002 ceph-mon[66732]: pgmap v285: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:31:23.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:31:23 smithi099 ceph-mon[56712]: pgmap v286: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:31:23.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:31:23 smithi002 ceph-mon[63804]: pgmap v286: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:31:23.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:31:23 smithi002 ceph-mon[66732]: pgmap v286: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:31:24.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:31:24 smithi099 ceph-mon[56712]: from='client.44938 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:31:24.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:31:24 smithi002 ceph-mon[63804]: from='client.44938 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:31:24.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:31:24 smithi002 ceph-mon[66732]: from='client.44938 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:31:25.868 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:31:25 smithi099 ceph-mon[56712]: pgmap v287: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:31:25.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:31:25 smithi002 ceph-mon[63804]: pgmap v287: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:31:25.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:31:25 smithi002 ceph-mon[66732]: pgmap v287: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:31:26.979 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:31:26 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: ::ffff:172.21.15.99 - - [17/Sep/2024:00:31:26] "GET /metrics HTTP/1.1" 200 38704 "" "Prometheus/2.43.0" 2024-09-17T00:31:27.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:31:27 smithi099 ceph-mon[56712]: pgmap v288: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-17T00:31:27.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:31:27 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T00:31:27.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:31:27 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:31:27.968 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:31:27 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:31:27.968 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:31:27 smithi099 ceph-mon[56712]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:31:27.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:31:27 smithi002 ceph-mon[63804]: pgmap v288: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-17T00:31:27.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:31:27 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T00:31:27.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:31:27 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:31:27.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:31:27 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:31:27.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:31:27 smithi002 ceph-mon[63804]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:31:27.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:31:27 smithi002 ceph-mon[66732]: pgmap v288: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-17T00:31:27.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:31:27 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T00:31:27.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:31:27 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-17T00:31:27.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:31:27 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-17T00:31:27.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:31:27 smithi002 ceph-mon[66732]: from='mgr.25256 ' entity='mgr.y' 2024-09-17T00:31:28.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:31:28 smithi099 ceph-mon[56712]: pgmap v289: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 975 B/s rd, 0 op/s 2024-09-17T00:31:28.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:31:28 smithi099 ceph-mon[56712]: pgmap v290: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:31:28.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:31:28 smithi002 ceph-mon[66732]: pgmap v289: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 975 B/s rd, 0 op/s 2024-09-17T00:31:28.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:31:28 smithi002 ceph-mon[66732]: pgmap v290: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:31:28.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:31:28 smithi002 ceph-mon[63804]: pgmap v289: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 975 B/s rd, 0 op/s 2024-09-17T00:31:28.980 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:31:28 smithi002 ceph-mon[63804]: pgmap v290: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:31:30.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:31:30 smithi099 ceph-mon[56712]: pgmap v291: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:31:30.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:31:30 smithi002 ceph-mon[66732]: pgmap v291: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:31:30.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:31:30 smithi002 ceph-mon[63804]: pgmap v291: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:31:32.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:31:32 smithi099 ceph-mon[56712]: pgmap v292: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:31:32.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:31:32 smithi002 ceph-mon[66732]: pgmap v292: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:31:32.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:31:32 smithi002 ceph-mon[63804]: pgmap v292: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:31:34.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:31:34 smithi099 ceph-mon[56712]: from='client.44938 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:31:34.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:31:34 smithi099 ceph-mon[56712]: pgmap v293: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 602 B/s rd, 0 op/s 2024-09-17T00:31:34.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:31:34 smithi002 ceph-mon[63804]: from='client.44938 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:31:34.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:31:34 smithi002 ceph-mon[63804]: pgmap v293: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 602 B/s rd, 0 op/s 2024-09-17T00:31:34.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:31:34 smithi002 ceph-mon[66732]: from='client.44938 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:31:34.980 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:31:34 smithi002 ceph-mon[66732]: pgmap v293: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 602 B/s rd, 0 op/s 2024-09-17T00:31:36.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:31:36 smithi099 ceph-mon[56712]: pgmap v294: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:31:36.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:31:36 smithi002 ceph-mon[66732]: pgmap v294: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:31:36.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:31:36 smithi002 ceph-mon[63804]: pgmap v294: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:31:36.980 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:31:36 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: ::ffff:172.21.15.99 - - [17/Sep/2024:00:31:36] "GET /metrics HTTP/1.1" 200 38702 "" "Prometheus/2.43.0" 2024-09-17T00:31:38.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:31:38 smithi099 ceph-mon[56712]: pgmap v295: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-17T00:31:38.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:31:38 smithi002 ceph-mon[66732]: pgmap v295: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-17T00:31:38.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:31:38 smithi002 ceph-mon[63804]: pgmap v295: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-17T00:31:40.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:31:40 smithi099 ceph-mon[56712]: pgmap v296: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:31:40.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:31:40 smithi002 ceph-mon[66732]: pgmap v296: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:31:40.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:31:40 smithi002 ceph-mon[63804]: pgmap v296: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:31:42.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:31:42 smithi099 ceph-mon[56712]: pgmap v297: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:31:42.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:31:42 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T00:31:42.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:31:42 smithi002 ceph-mon[63804]: pgmap v297: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:31:42.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:31:42 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T00:31:42.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:31:42 smithi002 ceph-mon[66732]: pgmap v297: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:31:42.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:31:42 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T00:31:44.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:31:44 smithi099 ceph-mon[56712]: from='client.44938 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:31:44.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:31:44 smithi099 ceph-mon[56712]: pgmap v298: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:31:44.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:31:44 smithi002 ceph-mon[63804]: from='client.44938 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:31:44.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:31:44 smithi002 ceph-mon[63804]: pgmap v298: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:31:44.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:31:44 smithi002 ceph-mon[66732]: from='client.44938 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:31:44.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:31:44 smithi002 ceph-mon[66732]: pgmap v298: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:31:45.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:31:45 smithi099 ceph-mon[56712]: pgmap v299: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:31:45.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:31:45 smithi002 ceph-mon[66732]: pgmap v299: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:31:45.979 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:31:45 smithi002 ceph-mon[63804]: pgmap v299: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:31:46.979 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:31:46 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: ::ffff:172.21.15.99 - - [17/Sep/2024:00:31:46] "GET /metrics HTTP/1.1" 200 38699 "" "Prometheus/2.43.0" 2024-09-17T00:31:48.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:31:48 smithi099 ceph-mon[56712]: pgmap v300: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:31:48.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:31:48 smithi002 ceph-mon[66732]: pgmap v300: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:31:48.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:31:48 smithi002 ceph-mon[63804]: pgmap v300: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:31:50.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:31:50 smithi099 ceph-mon[56712]: pgmap v301: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:31:50.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:31:50 smithi002 ceph-mon[66732]: pgmap v301: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:31:50.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:31:50 smithi002 ceph-mon[63804]: pgmap v301: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:31:52.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:31:52 smithi099 ceph-mon[56712]: pgmap v302: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:31:52.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:31:52 smithi002 ceph-mon[66732]: pgmap v302: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:31:52.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:31:52 smithi002 ceph-mon[63804]: pgmap v302: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:31:54.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:31:54 smithi099 ceph-mon[56712]: from='client.44938 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:31:54.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:31:54 smithi099 ceph-mon[56712]: pgmap v303: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:31:54.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:31:54 smithi002 ceph-mon[66732]: from='client.44938 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:31:54.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:31:54 smithi002 ceph-mon[66732]: pgmap v303: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:31:54.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:31:54 smithi002 ceph-mon[63804]: from='client.44938 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:31:54.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:31:54 smithi002 ceph-mon[63804]: pgmap v303: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:31:54.910 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3aad27a2-7489-11ef-bceb-c7b262605968 -e sha1=c2ddf9721ef254645201cdbabd9c8f49c30a586f -- bash -c 'ceph orch ps' 2024-09-17T00:31:56.268 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:31:56 smithi099 ceph-mon[56712]: pgmap v304: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:31:56.327 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:31:56 smithi002 ceph-mon[63804]: pgmap v304: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:31:56.327 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:31:56 smithi002 ceph-mon[66732]: pgmap v304: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:31:56.938 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:31:56 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: ::ffff:172.21.15.99 - - [17/Sep/2024:00:31:56] "GET /metrics HTTP/1.1" 200 38699 "" "Prometheus/2.43.0" 2024-09-17T00:31:57.186 INFO:teuthology.orchestra.run.smithi002.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-17T00:31:57.187 INFO:teuthology.orchestra.run.smithi002.stdout:alertmanager.a smithi002 *:9093,9094 running (9m) 91s ago 14m - - 0.25.0 c8568f914cd2 a21e13b3d677 2024-09-17T00:31:57.187 INFO:teuthology.orchestra.run.smithi002.stdout:grafana.a smithi099 *:3000 running (9m) 91s ago 14m 54.4M - 8.3.5 dad864ee21e9 bafb824a3c08 2024-09-17T00:31:57.187 INFO:teuthology.orchestra.run.smithi002.stdout:iscsi.foo.smithi002.vmouem smithi002 unknown 91s ago 13m - - 2024-09-17T00:31:57.187 INFO:teuthology.orchestra.run.smithi002.stdout:mgr.x smithi099 *:8443,9283,8765 running (8m) 91s ago 19m 443M - 19.1.1-330-gc2ddf972 ebb0615149de 6aa2ccef9ca0 2024-09-17T00:31:57.187 INFO:teuthology.orchestra.run.smithi002.stdout:mgr.y smithi002 *:8443,9283,8765 running (9m) 91s ago 20m - - 19.1.1-330-gc2ddf972 ebb0615149de 8838897e8449 2024-09-17T00:31:57.187 INFO:teuthology.orchestra.run.smithi002.stdout:mon.a smithi002 running (8m) 91s ago 21m - 2048M 19.1.1-330-gc2ddf972 ebb0615149de a035def53ef6 2024-09-17T00:31:57.187 INFO:teuthology.orchestra.run.smithi002.stdout:mon.b smithi099 running (7m) 91s ago 19m 46.0M 2048M 19.1.1-330-gc2ddf972 ebb0615149de aab3ab73a850 2024-09-17T00:31:57.187 INFO:teuthology.orchestra.run.smithi002.stdout:mon.c smithi002 running (7m) 91s ago 19m - 2048M 19.1.1-330-gc2ddf972 ebb0615149de 8a510255ae66 2024-09-17T00:31:57.187 INFO:teuthology.orchestra.run.smithi002.stdout:node-exporter.a smithi002 *:9100 running (9m) 91s ago 15m - - 1.5.0 0da6a335fe13 1030ae01bb32 2024-09-17T00:31:57.187 INFO:teuthology.orchestra.run.smithi002.stdout:node-exporter.b smithi099 *:9100 running (9m) 91s ago 15m 16.5M - 1.5.0 0da6a335fe13 01fcd35dca8a 2024-09-17T00:31:57.188 INFO:teuthology.orchestra.run.smithi002.stdout:osd.0 smithi002 running (6m) 91s ago 18m - 2737M 19.1.1-330-gc2ddf972 ebb0615149de 8345719cb6f0 2024-09-17T00:31:57.188 INFO:teuthology.orchestra.run.smithi002.stdout:osd.1 smithi002 running (6m) 91s ago 18m - 2737M 19.1.1-330-gc2ddf972 ebb0615149de 3dbc23d57c53 2024-09-17T00:31:57.188 INFO:teuthology.orchestra.run.smithi002.stdout:osd.2 smithi002 running (5m) 91s ago 18m - 2737M 19.1.1-330-gc2ddf972 ebb0615149de d21401b6763b 2024-09-17T00:31:57.188 INFO:teuthology.orchestra.run.smithi002.stdout:osd.3 smithi002 running (4m) 91s ago 17m - 2737M 19.1.1-330-gc2ddf972 ebb0615149de 980f2a25d096 2024-09-17T00:31:57.188 INFO:teuthology.orchestra.run.smithi002.stdout:osd.4 smithi099 running (4m) 91s ago 17m 59.0M 2992M 19.1.1-330-gc2ddf972 ebb0615149de f644e5be4db8 2024-09-17T00:31:57.188 INFO:teuthology.orchestra.run.smithi002.stdout:osd.5 smithi099 running (3m) 91s ago 17m 57.5M 2992M 19.1.1-330-gc2ddf972 ebb0615149de 2df0d88d16ce 2024-09-17T00:31:57.188 INFO:teuthology.orchestra.run.smithi002.stdout:osd.6 smithi099 running (2m) 91s ago 16m 50.4M 2992M 19.1.1-330-gc2ddf972 ebb0615149de 7462bdbdf29b 2024-09-17T00:31:57.188 INFO:teuthology.orchestra.run.smithi002.stdout:osd.7 smithi099 running (2m) 91s ago 16m 62.1M 2992M 19.1.1-330-gc2ddf972 ebb0615149de 9821b8012a7e 2024-09-17T00:31:57.188 INFO:teuthology.orchestra.run.smithi002.stdout:prometheus.a smithi099 *:9095 running (8m) 91s ago 15m 67.4M - 2.43.0 a07b618ecd1d bf32c2ae11dc 2024-09-17T00:31:57.188 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.foo.smithi002.uwdhcs smithi002 *:8000 running (2m) 91s ago 13m - - 19.1.1-330-gc2ddf972 ebb0615149de 19fa1132d770 2024-09-17T00:31:57.188 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.foo.smithi099.utzfap smithi099 *:8000 running (115s) 91s ago 13m 76.4M - 19.1.1-330-gc2ddf972 ebb0615149de b56c819bfbb8 2024-09-17T00:31:57.189 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.smpl.smithi002.rqnxii smithi002 *:80 running (2m) 91s ago 13m - - 19.1.1-330-gc2ddf972 ebb0615149de 50f19f6bf716 2024-09-17T00:31:57.189 INFO:teuthology.orchestra.run.smithi002.stdout:rgw.smpl.smithi099.nzvkpm smithi099 *:80 running (109s) 91s ago 13m 77.0M - 19.1.1-330-gc2ddf972 ebb0615149de 4256e3820d2d 2024-09-17T00:31:57.200 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:31:57 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T00:31:57.200 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:31:57 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T00:31:57.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:31:57 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T00:31:57.963 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3aad27a2-7489-11ef-bceb-c7b262605968 -e sha1=c2ddf9721ef254645201cdbabd9c8f49c30a586f -- bash -c 'ceph versions' 2024-09-17T00:31:58.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:31:58 smithi002 ceph-mon[63804]: pgmap v305: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:31:58.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:31:58 smithi002 ceph-mon[63804]: from='client.44977 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:31:58.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:31:58 smithi002 ceph-mon[66732]: pgmap v305: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:31:58.230 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:31:58 smithi002 ceph-mon[66732]: from='client.44977 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:31:58.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:31:58 smithi099 ceph-mon[56712]: pgmap v305: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:31:58.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:31:58 smithi099 ceph-mon[56712]: from='client.44977 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:32:00.262 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-09-17T00:32:00.262 INFO:teuthology.orchestra.run.smithi002.stdout: "mon": { 2024-09-17T00:32:00.262 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 19.1.1-330-gc2ddf972 (c2ddf9721ef254645201cdbabd9c8f49c30a586f) squid (stable)": 3 2024-09-17T00:32:00.262 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-09-17T00:32:00.262 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr": { 2024-09-17T00:32:00.262 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 19.1.1-330-gc2ddf972 (c2ddf9721ef254645201cdbabd9c8f49c30a586f) squid (stable)": 2 2024-09-17T00:32:00.262 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-09-17T00:32:00.262 INFO:teuthology.orchestra.run.smithi002.stdout: "osd": { 2024-09-17T00:32:00.263 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 19.1.1-330-gc2ddf972 (c2ddf9721ef254645201cdbabd9c8f49c30a586f) squid (stable)": 8 2024-09-17T00:32:00.263 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-09-17T00:32:00.263 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw": { 2024-09-17T00:32:00.263 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 19.1.1-330-gc2ddf972 (c2ddf9721ef254645201cdbabd9c8f49c30a586f) squid (stable)": 4 2024-09-17T00:32:00.263 INFO:teuthology.orchestra.run.smithi002.stdout: }, 2024-09-17T00:32:00.263 INFO:teuthology.orchestra.run.smithi002.stdout: "overall": { 2024-09-17T00:32:00.263 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 19.1.1-330-gc2ddf972 (c2ddf9721ef254645201cdbabd9c8f49c30a586f) squid (stable)": 17 2024-09-17T00:32:00.263 INFO:teuthology.orchestra.run.smithi002.stdout: } 2024-09-17T00:32:00.263 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-09-17T00:32:00.275 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:32:00 smithi002 ceph-mon[66732]: pgmap v306: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:32:00.275 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:32:00 smithi002 ceph-mon[63804]: pgmap v306: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:32:00.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:32:00 smithi099 ceph-mon[56712]: pgmap v306: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:32:01.004 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3aad27a2-7489-11ef-bceb-c7b262605968 -e sha1=c2ddf9721ef254645201cdbabd9c8f49c30a586f -- bash -c 'ceph orch upgrade status' 2024-09-17T00:32:01.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:32:01 smithi002 ceph-mon[66732]: from='client.? 172.21.15.2:0/768954716' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:32:01.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:32:01 smithi002 ceph-mon[63804]: from='client.? 172.21.15.2:0/768954716' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:32:01.468 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:32:01 smithi099 ceph-mon[56712]: from='client.? 172.21.15.2:0/768954716' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:32:02.373 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:32:02 smithi002 ceph-mon[66732]: pgmap v307: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:32:02.373 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:32:02 smithi002 ceph-mon[63804]: pgmap v307: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:32:02.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:32:02 smithi099 ceph-mon[56712]: pgmap v307: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:32:03.146 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-09-17T00:32:03.146 INFO:teuthology.orchestra.run.smithi002.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f", 2024-09-17T00:32:03.146 INFO:teuthology.orchestra.run.smithi002.stdout: "in_progress": true, 2024-09-17T00:32:03.146 INFO:teuthology.orchestra.run.smithi002.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-17T00:32:03.146 INFO:teuthology.orchestra.run.smithi002.stdout: "services_complete": [ 2024-09-17T00:32:03.146 INFO:teuthology.orchestra.run.smithi002.stdout: "osd", 2024-09-17T00:32:03.146 INFO:teuthology.orchestra.run.smithi002.stdout: "mgr", 2024-09-17T00:32:03.146 INFO:teuthology.orchestra.run.smithi002.stdout: "mon", 2024-09-17T00:32:03.147 INFO:teuthology.orchestra.run.smithi002.stdout: "rgw" 2024-09-17T00:32:03.147 INFO:teuthology.orchestra.run.smithi002.stdout: ], 2024-09-17T00:32:03.147 INFO:teuthology.orchestra.run.smithi002.stdout: "progress": "17/23 daemons upgraded", 2024-09-17T00:32:03.147 INFO:teuthology.orchestra.run.smithi002.stdout: "message": "Error: UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi002.vmouem on host smithi002 failed.", 2024-09-17T00:32:03.147 INFO:teuthology.orchestra.run.smithi002.stdout: "is_paused": true 2024-09-17T00:32:03.147 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-09-17T00:32:03.809 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3aad27a2-7489-11ef-bceb-c7b262605968 -e sha1=c2ddf9721ef254645201cdbabd9c8f49c30a586f -- bash -c 'ceph health detail' 2024-09-17T00:32:04.392 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:32:04 smithi002 ceph-mon[66732]: from='client.44938 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:32:04.392 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:32:04 smithi002 ceph-mon[66732]: pgmap v308: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:32:04.392 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:32:04 smithi002 ceph-mon[66732]: from='client.44989 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:32:04.392 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:32:04 smithi002 ceph-mon[63804]: from='client.44938 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:32:04.393 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:32:04 smithi002 ceph-mon[63804]: pgmap v308: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:32:04.393 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:32:04 smithi002 ceph-mon[63804]: from='client.44989 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:32:04.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:32:04 smithi099 ceph-mon[56712]: from='client.44938 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:32:04.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:32:04 smithi099 ceph-mon[56712]: pgmap v308: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:32:04.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:32:04 smithi099 ceph-mon[56712]: from='client.44989 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:32:06.192 INFO:teuthology.orchestra.run.smithi002.stdout:HEALTH_WARN 1 failed cephadm daemon(s); Upgrading daemon iscsi.foo.smithi002.vmouem on host smithi002 failed. 2024-09-17T00:32:06.193 INFO:teuthology.orchestra.run.smithi002.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-09-17T00:32:06.193 INFO:teuthology.orchestra.run.smithi002.stdout: daemon iscsi.foo.smithi002.vmouem on smithi002 is in unknown state 2024-09-17T00:32:06.193 INFO:teuthology.orchestra.run.smithi002.stdout:[WRN] UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi002.vmouem on host smithi002 failed. 2024-09-17T00:32:06.193 INFO:teuthology.orchestra.run.smithi002.stdout: Upgrade daemon: iscsi.foo.smithi002.vmouem: cephadm exited with an error code: 1, stderr: Redeploy daemon iscsi.foo.smithi002.vmouem ... 2024-09-17T00:32:06.193 INFO:teuthology.orchestra.run.smithi002.stdout:Creating ceph-iscsi config... 2024-09-17T00:32:06.193 INFO:teuthology.orchestra.run.smithi002.stdout:Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/iscsi-gateway.cfg 2024-09-17T00:32:06.193 INFO:teuthology.orchestra.run.smithi002.stdout:Write file: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/iscsi.foo.smithi002.vmouem/tcmu-runner-entrypoint.sh 2024-09-17T00:32:06.193 INFO:teuthology.orchestra.run.smithi002.stdout:Failed to trim old cgroups /sys/fs/cgroup/system.slice/system-ceph\x2d3aad27a2\x2d7489\x2d11ef\x2dbceb\x2dc7b262605968.slice/ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service 2024-09-17T00:32:06.193 INFO:teuthology.orchestra.run.smithi002.stdout:Non-zero exit code 1 from systemctl start ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem 2024-09-17T00:32:06.193 INFO:teuthology.orchestra.run.smithi002.stdout:systemctl: stderr Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:32:06.193 INFO:teuthology.orchestra.run.smithi002.stdout:systemctl: stderr See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:32:06.193 INFO:teuthology.orchestra.run.smithi002.stdout:Traceback (most recent call last): 2024-09-17T00:32:06.194 INFO:teuthology.orchestra.run.smithi002.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-17T00:32:06.194 INFO:teuthology.orchestra.run.smithi002.stdout: return _run_code(code, main_globals, None, 2024-09-17T00:32:06.194 INFO:teuthology.orchestra.run.smithi002.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-17T00:32:06.194 INFO:teuthology.orchestra.run.smithi002.stdout: exec(code, run_globals) 2024-09-17T00:32:06.194 INFO:teuthology.orchestra.run.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5579, in 2024-09-17T00:32:06.194 INFO:teuthology.orchestra.run.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 5567, in main 2024-09-17T00:32:06.194 INFO:teuthology.orchestra.run.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3049, in command_deploy_from 2024-09-17T00:32:06.194 INFO:teuthology.orchestra.run.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3084, in _common_deploy 2024-09-17T00:32:06.194 INFO:teuthology.orchestra.run.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 3104, in _deploy_daemon_container 2024-09-17T00:32:06.194 INFO:teuthology.orchestra.run.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 1088, in deploy_daemon 2024-09-17T00:32:06.194 INFO:teuthology.orchestra.run.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/__main__.py", line 1224, in deploy_daemon_units 2024-09-17T00:32:06.194 INFO:teuthology.orchestra.run.smithi002.stdout: File "/tmp/tmpnroy8d0w.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-17T00:32:06.194 INFO:teuthology.orchestra.run.smithi002.stdout:RuntimeError: Failed command: systemctl start ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem: Job for ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service failed because the control process exited with error code. 2024-09-17T00:32:06.194 INFO:teuthology.orchestra.run.smithi002.stdout:See "systemctl status ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" and "journalctl -xeu ceph-3aad27a2-7489-11ef-bceb-c7b262605968@iscsi.foo.smithi002.vmouem.service" for details. 2024-09-17T00:32:06.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:32:06 smithi099 ceph-mon[56712]: pgmap v309: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:32:06.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:32:06 smithi002 ceph-mon[66732]: pgmap v309: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:32:06.479 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:32:06 smithi002 ceph-mon[63804]: pgmap v309: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:32:06.921 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:32:06 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: ::ffff:172.21.15.99 - - [17/Sep/2024:00:32:06] "GET /metrics HTTP/1.1" 200 38704 "" "Prometheus/2.43.0" 2024-09-17T00:32:06.952 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3aad27a2-7489-11ef-bceb-c7b262605968 -e sha1=c2ddf9721ef254645201cdbabd9c8f49c30a586f -- bash -c 'ceph versions | jq -e '"'"'.overall | length == 1'"'"'' 2024-09-17T00:32:07.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:32:07 smithi002 ceph-mon[66732]: from='client.? 172.21.15.2:0/3111733786' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T00:32:07.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:32:07 smithi002 ceph-mon[63804]: from='client.? 172.21.15.2:0/3111733786' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T00:32:07.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:32:07 smithi099 ceph-mon[56712]: from='client.? 172.21.15.2:0/3111733786' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-17T00:32:08.312 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:32:08 smithi002 ceph-mon[66732]: pgmap v310: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:32:08.328 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:32:08 smithi002 ceph-mon[63804]: pgmap v310: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:32:08.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:32:08 smithi099 ceph-mon[56712]: pgmap v310: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:32:09.188 INFO:teuthology.orchestra.run.smithi002.stdout:true 2024-09-17T00:32:09.844 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3aad27a2-7489-11ef-bceb-c7b262605968 -e sha1=c2ddf9721ef254645201cdbabd9c8f49c30a586f -- bash -c 'ceph versions | jq -e '"'"'.overall | keys'"'"' | grep $sha1' 2024-09-17T00:32:10.153 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:32:10 smithi002 ceph-mon[66732]: pgmap v311: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:32:10.153 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:32:10 smithi002 ceph-mon[66732]: from='client.? 172.21.15.2:0/2432865960' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:32:10.154 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:32:10 smithi002 ceph-mon[63804]: pgmap v311: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:32:10.154 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:32:10 smithi002 ceph-mon[63804]: from='client.? 172.21.15.2:0/2432865960' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:32:10.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:32:10 smithi099 ceph-mon[56712]: pgmap v311: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:32:10.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:32:10 smithi099 ceph-mon[56712]: from='client.? 172.21.15.2:0/2432865960' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:32:12.026 INFO:teuthology.orchestra.run.smithi002.stdout: "ceph version 19.1.1-330-gc2ddf972 (c2ddf9721ef254645201cdbabd9c8f49c30a586f) squid (stable)" 2024-09-17T00:32:12.309 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:32:12 smithi002 ceph-mon[63804]: pgmap v312: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:32:12.309 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:32:12 smithi002 ceph-mon[63804]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T00:32:12.309 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:32:12 smithi002 ceph-mon[63804]: from='client.? 172.21.15.2:0/3438259282' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:32:12.310 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:32:12 smithi002 ceph-mon[66732]: pgmap v312: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:32:12.310 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:32:12 smithi002 ceph-mon[66732]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T00:32:12.310 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:32:12 smithi002 ceph-mon[66732]: from='client.? 172.21.15.2:0/3438259282' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:32:12.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:32:12 smithi099 ceph-mon[56712]: pgmap v312: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:32:12.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:32:12 smithi099 ceph-mon[56712]: from='mgr.25256 172.21.15.2:0/3141115098' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-17T00:32:12.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:32:12 smithi099 ceph-mon[56712]: from='client.? 172.21.15.2:0/3438259282' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-17T00:32:12.651 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3aad27a2-7489-11ef-bceb-c7b262605968 -e sha1=c2ddf9721ef254645201cdbabd9c8f49c30a586f -- bash -c 'ceph orch ls | grep '"'"'^osd '"'"'' 2024-09-17T00:32:14.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:32:14 smithi002 ceph-mon[66732]: from='client.44938 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:32:14.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:32:14 smithi002 ceph-mon[66732]: pgmap v313: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:32:14.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:32:14 smithi002 ceph-mon[63804]: from='client.44938 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:32:14.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:32:14 smithi002 ceph-mon[63804]: pgmap v313: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:32:14.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:32:14 smithi099 ceph-mon[56712]: from='client.44938 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:32:14.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:32:14 smithi099 ceph-mon[56712]: pgmap v313: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:32:14.731 INFO:teuthology.orchestra.run.smithi002.stdout:osd 8 109s ago - 2024-09-17T00:32:15.478 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-09-17T00:32:15.490 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi002.front.sepia.ceph.com 2024-09-17T00:32:15.491 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3aad27a2-7489-11ef-bceb-c7b262605968 -- bash -c 'ceph orch upgrade ls' 2024-09-17T00:32:16.060 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:32:16 smithi002 ceph-mon[66732]: from='client.35118 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:32:16.060 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:32:16 smithi002 ceph-mon[66732]: pgmap v314: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:32:16.060 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:32:16 smithi002 ceph-mon[63804]: from='client.35118 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:32:16.060 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:32:16 smithi002 ceph-mon[63804]: pgmap v314: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:32:16.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:32:16 smithi099 ceph-mon[56712]: from='client.35118 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:32:16.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:32:16 smithi099 ceph-mon[56712]: pgmap v314: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:32:16.923 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:32:16 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: ::ffff:172.21.15.99 - - [17/Sep/2024:00:32:16] "GET /metrics HTTP/1.1" 200 38700 "" "Prometheus/2.43.0" 2024-09-17T00:32:17.988 INFO:teuthology.orchestra.run.smithi002.stdout:{ 2024-09-17T00:32:17.989 INFO:teuthology.orchestra.run.smithi002.stdout: "image": "quay.io/ceph/ceph", 2024-09-17T00:32:17.989 INFO:teuthology.orchestra.run.smithi002.stdout: "registry": "quay.io", 2024-09-17T00:32:17.989 INFO:teuthology.orchestra.run.smithi002.stdout: "bare_image": "ceph/ceph", 2024-09-17T00:32:17.989 INFO:teuthology.orchestra.run.smithi002.stdout: "versions": [ 2024-09-17T00:32:17.989 INFO:teuthology.orchestra.run.smithi002.stdout: "19.1.1", 2024-09-17T00:32:17.989 INFO:teuthology.orchestra.run.smithi002.stdout: "19.1.0" 2024-09-17T00:32:17.989 INFO:teuthology.orchestra.run.smithi002.stdout: ] 2024-09-17T00:32:17.989 INFO:teuthology.orchestra.run.smithi002.stdout:} 2024-09-17T00:32:18.363 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:32:18 smithi002 ceph-mon[63804]: pgmap v315: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:32:18.363 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:32:18 smithi002 ceph-mon[66732]: pgmap v315: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:32:18.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:32:18 smithi099 ceph-mon[56712]: pgmap v315: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:32:18.773 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3aad27a2-7489-11ef-bceb-c7b262605968 -- bash -c 'ceph orch upgrade ls --image quay.io/ceph/ceph --show-all-versions | grep 16.2.0' 2024-09-17T00:32:19.391 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:32:19 smithi002 ceph-mon[66732]: from='client.35124 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:32:19.392 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:32:19 smithi002 ceph-mon[63804]: from='client.35124 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:32:19.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:32:19 smithi099 ceph-mon[56712]: from='client.35124 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:32:20.413 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:32:20 smithi002 ceph-mon[63804]: pgmap v316: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:32:20.413 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:32:20 smithi002 ceph-mon[66732]: pgmap v316: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:32:20.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:32:20 smithi099 ceph-mon[56712]: pgmap v316: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-17T00:32:21.253 INFO:teuthology.orchestra.run.smithi002.stdout: "16.2.0", 2024-09-17T00:32:21.976 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3aad27a2-7489-11ef-bceb-c7b262605968 -- bash -c 'ceph orch upgrade ls --image quay.io/ceph/ceph --tags | grep v16.2.2' 2024-09-17T00:32:22.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:32:22 smithi002 ceph-mon[66732]: from='client.35130 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "show_all_versions": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:32:22.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:32:22 smithi002 ceph-mon[66732]: pgmap v317: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:32:22.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:32:22 smithi002 ceph-mon[63804]: from='client.35130 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "show_all_versions": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:32:22.229 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:32:22 smithi002 ceph-mon[63804]: pgmap v317: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:32:22.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:32:22 smithi099 ceph-mon[56712]: from='client.35130 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "show_all_versions": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:32:22.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:32:22 smithi099 ceph-mon[56712]: pgmap v317: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:32:24.388 INFO:teuthology.orchestra.run.smithi002.stdout: "v16.2.2", 2024-09-17T00:32:24.389 INFO:teuthology.orchestra.run.smithi002.stdout: "v16.2.2-20210505", 2024-09-17T00:32:24.389 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:32:24 smithi002 ceph-mon[66732]: from='client.44938 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:32:24.389 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:32:24 smithi002 ceph-mon[66732]: pgmap v318: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:32:24.389 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:32:24 smithi002 ceph-mon[63804]: from='client.44938 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:32:24.389 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:32:24 smithi002 ceph-mon[63804]: pgmap v318: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:32:24.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:32:24 smithi099 ceph-mon[56712]: from='client.44938 -' entity='client.iscsi.foo.smithi002.vmouem' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-17T00:32:24.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:32:24 smithi099 ceph-mon[56712]: pgmap v318: 161 pgs: 161 active+clean; 585 KiB data, 286 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-17T00:32:25.043 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-09-17T00:32:25.056 INFO:tasks.cephadm:Teardown begin 2024-09-17T00:32:25.057 DEBUG:teuthology.orchestra.run.smithi002:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-17T00:32:25.088 DEBUG:teuthology.orchestra.run.smithi099:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-17T00:32:25.120 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-09-17T00:32:25.120 DEBUG:teuthology.orchestra.run.smithi002:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-09-17T00:32:25.145 DEBUG:teuthology.orchestra.run.smithi099:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-09-17T00:32:25.177 INFO:tasks.cephadm:Stopping all daemons... 2024-09-17T00:32:25.178 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-09-17T00:32:25.178 DEBUG:teuthology.orchestra.run.smithi002:> sudo systemctl stop ceph-3aad27a2-7489-11ef-bceb-c7b262605968@mon.a 2024-09-17T00:32:25.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:32:25 smithi099 ceph-mon[56712]: from='client.45031 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "tags": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:32:25.479 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:32:25 smithi002 ceph-mon[66732]: from='client.45031 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "tags": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:32:25.480 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:32:25 smithi002 ceph-mon[63804]: from='client.45031 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "tags": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-17T00:32:25.480 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:32:25 smithi002 systemd[1]: Stopping Ceph mon.a for 3aad27a2-7489-11ef-bceb-c7b262605968... 2024-09-17T00:32:25.790 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:32:25 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mon-a[63800]: 2024-09-17T00:32:25.517+0000 7fa98462b640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-17T00:32:25.790 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:32:25 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mon-a[63800]: 2024-09-17T00:32:25.517+0000 7fa98462b640 -1 mon.a@0(leader) e4 *** Got Signal Terminated *** 2024-09-17T00:32:26.064 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:32:25 smithi002 podman[94664]: 2024-09-17 00:32:25.793439809 +0000 UTC m=+0.408251824 container died a035def53ef68b141bb6821552dcaaad986ad917acefceef91af915e462c773c (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mon-a, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=squid-c2ddf97, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-09-17T00:32:26.455 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:32:26 smithi002 podman[94664]: 2024-09-17 00:32:26.173422462 +0000 UTC m=+0.788234482 container cleanup a035def53ef68b141bb6821552dcaaad986ad917acefceef91af915e462c773c (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mon-a, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=squid-c2ddf97, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-09-17T00:32:26.455 INFO:journalctl@ceph.mon.a.smithi002.stdout:Sep 17 00:32:26 smithi002 bash[94664]: ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mon-a 2024-09-17T00:32:26.610 DEBUG:teuthology.orchestra.run.smithi002:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3aad27a2-7489-11ef-bceb-c7b262605968@mon.a.service' 2024-09-17T00:32:26.656 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-17T00:32:26.656 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-09-17T00:32:26.656 INFO:tasks.cephadm.mon.b:Stopping mon.c... 2024-09-17T00:32:26.656 DEBUG:teuthology.orchestra.run.smithi002:> sudo systemctl stop ceph-3aad27a2-7489-11ef-bceb-c7b262605968@mon.c 2024-09-17T00:32:26.709 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:32:26 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: ::ffff:172.21.15.99 - - [17/Sep/2024:00:32:26] "GET /metrics HTTP/1.1" 200 38700 "" "Prometheus/2.43.0" 2024-09-17T00:32:26.967 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:32:26 smithi002 systemd[1]: Stopping Ceph mon.c for 3aad27a2-7489-11ef-bceb-c7b262605968... 2024-09-17T00:32:27.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:32:26 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mon-c[66728]: 2024-09-17T00:32:26.969+0000 7f416a1b7640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-17T00:32:27.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:32:26 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mon-c[66728]: 2024-09-17T00:32:26.969+0000 7f416a1b7640 -1 mon.c@1(peon) e4 *** Got Signal Terminated *** 2024-09-17T00:32:27.229 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:32:27 smithi002 podman[94785]: 2024-09-17 00:32:27.143166383 +0000 UTC m=+0.284767608 container died 8a510255ae66ca7c1a4d698b4e72ce9f3d0173536a2369511f9fd9c47ba44b7c (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mon-c, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=squid-c2ddf97, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD) 2024-09-17T00:32:27.230 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:32:26 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: [17/Sep/2024:00:32:26] ENGINE Bus STOPPING 2024-09-17T00:32:27.230 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:32:27 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: [17/Sep/2024:00:32:27] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-17T00:32:27.230 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:32:27 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: [17/Sep/2024:00:32:27] ENGINE Bus STOPPED 2024-09-17T00:32:27.230 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:32:27 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: [17/Sep/2024:00:32:27] ENGINE Bus STARTING 2024-09-17T00:32:27.559 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:32:27 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: [17/Sep/2024:00:32:27] ENGINE Serving on http://:::9283 2024-09-17T00:32:27.559 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:32:27 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: [17/Sep/2024:00:32:27] ENGINE Bus STARTED 2024-09-17T00:32:27.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:32:27 smithi002 podman[94785]: 2024-09-17 00:32:27.561945705 +0000 UTC m=+0.703546918 container cleanup 8a510255ae66ca7c1a4d698b4e72ce9f3d0173536a2369511f9fd9c47ba44b7c (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mon-c, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, RELEASE=squid-c2ddf97, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-09-17T00:32:27.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:32:27 smithi002 bash[94785]: ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mon-c 2024-09-17T00:32:27.979 INFO:journalctl@ceph.mon.c.smithi002.stdout:Sep 17 00:32:27 smithi002 podman[94863]: 2024-09-17 00:32:27.779852255 +0000 UTC m=+0.639220670 container remove 8a510255ae66ca7c1a4d698b4e72ce9f3d0173536a2369511f9fd9c47ba44b7c (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mon-c, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, RELEASE=squid-c2ddf97) 2024-09-17T00:32:28.048 DEBUG:teuthology.orchestra.run.smithi002:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3aad27a2-7489-11ef-bceb-c7b262605968@mon.c.service' 2024-09-17T00:32:28.085 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-17T00:32:28.085 INFO:tasks.cephadm.mon.b:Stopped mon.c 2024-09-17T00:32:28.085 INFO:tasks.cephadm.mon.b:Stopping mon.b... 2024-09-17T00:32:28.085 DEBUG:teuthology.orchestra.run.smithi099:> sudo systemctl stop ceph-3aad27a2-7489-11ef-bceb-c7b262605968@mon.b 2024-09-17T00:32:28.467 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:32:28 smithi099 systemd[1]: Stopping Ceph mon.b for 3aad27a2-7489-11ef-bceb-c7b262605968... 2024-09-17T00:32:28.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:32:28 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mon-b[56708]: 2024-09-17T00:32:28.579+0000 7ffb7bd20640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-17T00:32:28.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:32:28 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mon-b[56708]: 2024-09-17T00:32:28.579+0000 7ffb7bd20640 -1 mon.b@2(peon) e4 *** Got Signal Terminated *** 2024-09-17T00:32:28.967 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:32:28 smithi099 podman[75124]: 2024-09-17 00:32:28.691730344 +0000 UTC m=+0.398138142 container died aab3ab73a850570e9284d848e27b6d1efd9bba4063d67a8b5aa8413115306edf (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mon-b, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=squid-c2ddf97, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2) 2024-09-17T00:32:29.311 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:32:29 smithi099 podman[75124]: 2024-09-17 00:32:29.046115961 +0000 UTC m=+0.752523740 container cleanup aab3ab73a850570e9284d848e27b6d1efd9bba4063d67a8b5aa8413115306edf (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mon-b, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, RELEASE=squid-c2ddf97, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731) 2024-09-17T00:32:29.311 INFO:journalctl@ceph.mon.b.smithi099.stdout:Sep 17 00:32:29 smithi099 bash[75124]: ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mon-b 2024-09-17T00:32:29.470 DEBUG:teuthology.orchestra.run.smithi099:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3aad27a2-7489-11ef-bceb-c7b262605968@mon.b.service' 2024-09-17T00:32:29.514 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-17T00:32:29.514 INFO:tasks.cephadm.mon.b:Stopped mon.b 2024-09-17T00:32:29.515 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-09-17T00:32:29.515 DEBUG:teuthology.orchestra.run.smithi002:> sudo systemctl stop ceph-3aad27a2-7489-11ef-bceb-c7b262605968@mgr.y 2024-09-17T00:32:29.838 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:32:29 smithi002 systemd[1]: Stopping Ceph mgr.y for 3aad27a2-7489-11ef-bceb-c7b262605968... 2024-09-17T00:32:30.172 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:32:29 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y[59216]: 2024-09-17T00:32:29.840+0000 7f184d15f640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.y -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-17T00:32:30.172 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:32:29 smithi002 podman[95006]: 2024-09-17 00:32:29.882096821 +0000 UTC m=+0.162385642 container died 8838897e8449c9d8813bf341a6a719e644a65735e3df0b6e29c09bc8ec67ede2 (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, RELEASE=squid-c2ddf97) 2024-09-17T00:32:30.479 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:32:30 smithi002 podman[95006]: 2024-09-17 00:32:30.266099927 +0000 UTC m=+0.546388767 container cleanup 8838897e8449c9d8813bf341a6a719e644a65735e3df0b6e29c09bc8ec67ede2 (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=squid-c2ddf97, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, io.buildah.version=1.37.2) 2024-09-17T00:32:30.479 INFO:journalctl@ceph.mgr.y.smithi002.stdout:Sep 17 00:32:30 smithi002 bash[95006]: ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-y 2024-09-17T00:32:30.707 DEBUG:teuthology.orchestra.run.smithi002:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3aad27a2-7489-11ef-bceb-c7b262605968@mgr.y.service' 2024-09-17T00:32:30.744 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-17T00:32:30.744 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-09-17T00:32:30.744 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-09-17T00:32:30.745 DEBUG:teuthology.orchestra.run.smithi099:> sudo systemctl stop ceph-3aad27a2-7489-11ef-bceb-c7b262605968@mgr.x 2024-09-17T00:32:31.087 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:32:30 smithi099 systemd[1]: Stopping Ceph mgr.x for 3aad27a2-7489-11ef-bceb-c7b262605968... 2024-09-17T00:32:31.374 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:32:31 smithi099 podman[75247]: 2024-09-17 00:32:31.090284353 +0000 UTC m=+0.141783173 container died 6aa2ccef9ca06f456bc1be7be15b45be2b21b3f66a345e29eb5261d715dd9fc1 (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240731, RELEASE=squid-c2ddf97, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-17T00:32:31.717 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:32:31 smithi099 podman[75247]: 2024-09-17 00:32:31.459507116 +0000 UTC m=+0.511005923 container cleanup 6aa2ccef9ca06f456bc1be7be15b45be2b21b3f66a345e29eb5261d715dd9fc1 (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=squid-c2ddf97, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240731) 2024-09-17T00:32:31.717 INFO:journalctl@ceph.mgr.x.smithi099.stdout:Sep 17 00:32:31 smithi099 bash[75247]: ceph-3aad27a2-7489-11ef-bceb-c7b262605968-mgr-x 2024-09-17T00:32:31.883 DEBUG:teuthology.orchestra.run.smithi099:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3aad27a2-7489-11ef-bceb-c7b262605968@mgr.x.service' 2024-09-17T00:32:31.917 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-17T00:32:31.918 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-09-17T00:32:31.918 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-09-17T00:32:31.918 DEBUG:teuthology.orchestra.run.smithi002:> sudo systemctl stop ceph-3aad27a2-7489-11ef-bceb-c7b262605968@osd.0 2024-09-17T00:32:32.229 INFO:journalctl@ceph.osd.0.smithi002.stdout:Sep 17 00:32:31 smithi002 systemd[1]: Stopping Ceph osd.0 for 3aad27a2-7489-11ef-bceb-c7b262605968... 2024-09-17T00:32:32.729 INFO:journalctl@ceph.osd.0.smithi002.stdout:Sep 17 00:32:32 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-0[71240]: 2024-09-17T00:32:32.237+0000 7f9891dca640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-17T00:32:32.729 INFO:journalctl@ceph.osd.0.smithi002.stdout:Sep 17 00:32:32 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-0[71240]: 2024-09-17T00:32:32.237+0000 7f9891dca640 -1 osd.0 153 *** Got signal Terminated *** 2024-09-17T00:32:32.729 INFO:journalctl@ceph.osd.0.smithi002.stdout:Sep 17 00:32:32 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-0[71240]: 2024-09-17T00:32:32.237+0000 7f9891dca640 -1 osd.0 153 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-17T00:32:37.827 INFO:journalctl@ceph.osd.0.smithi002.stdout:Sep 17 00:32:37 smithi002 podman[95126]: 2024-09-17 00:32:37.541001354 +0000 UTC m=+5.416827961 container died 8345719cb6f0d21519d86b3f7fed22523f2ff0f9daa2ac0c3d61ccec8a7d4e8c (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=squid-c2ddf97, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-09-17T00:32:38.210 INFO:journalctl@ceph.osd.0.smithi002.stdout:Sep 17 00:32:37 smithi002 podman[95126]: 2024-09-17 00:32:37.920492272 +0000 UTC m=+5.796318870 container cleanup 8345719cb6f0d21519d86b3f7fed22523f2ff0f9daa2ac0c3d61ccec8a7d4e8c (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, RELEASE=squid-c2ddf97, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-09-17T00:32:38.210 INFO:journalctl@ceph.osd.0.smithi002.stdout:Sep 17 00:32:37 smithi002 bash[95126]: ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-0 2024-09-17T00:32:38.479 INFO:journalctl@ceph.osd.0.smithi002.stdout:Sep 17 00:32:38 smithi002 podman[95144]: 2024-09-17 00:32:38.213360358 +0000 UTC m=+0.661279009 container remove 8345719cb6f0d21519d86b3f7fed22523f2ff0f9daa2ac0c3d61ccec8a7d4e8c (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, RELEASE=squid-c2ddf97, org.label-schema.license=GPLv2) 2024-09-17T00:32:38.888 INFO:journalctl@ceph.osd.0.smithi002.stdout:Sep 17 00:32:38 smithi002 podman[95231]: 2024-09-17 00:32:38.450195254 +0000 UTC m=+0.016260994 image pull ebb0615149def4a3179c82c0cc47af75808300abb9f1142788e5e605a7ac0e5b quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f 2024-09-17T00:32:38.889 INFO:journalctl@ceph.osd.0.smithi002.stdout:Sep 17 00:32:38 smithi002 podman[95231]: 2024-09-17 00:32:38.555923312 +0000 UTC m=+0.121989032 container create 4a49b29c2fc94847b1c19e16a3d01a747b8c13f7816ccb180146cd2d4d9bda65 (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-0-deactivate, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-c2ddf97, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2) 2024-09-17T00:32:38.889 INFO:journalctl@ceph.osd.0.smithi002.stdout:Sep 17 00:32:38 smithi002 podman[95231]: 2024-09-17 00:32:38.792164152 +0000 UTC m=+0.358229892 container init 4a49b29c2fc94847b1c19e16a3d01a747b8c13f7816ccb180146cd2d4d9bda65 (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-0-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=squid-c2ddf97) 2024-09-17T00:32:38.889 INFO:journalctl@ceph.osd.0.smithi002.stdout:Sep 17 00:32:38 smithi002 podman[95231]: 2024-09-17 00:32:38.795551398 +0000 UTC m=+0.361617123 container start 4a49b29c2fc94847b1c19e16a3d01a747b8c13f7816ccb180146cd2d4d9bda65 (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-0-deactivate, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, RELEASE=squid-c2ddf97, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-09-17T00:32:38.889 INFO:journalctl@ceph.osd.0.smithi002.stdout:Sep 17 00:32:38 smithi002 podman[95231]: 2024-09-17 00:32:38.891997703 +0000 UTC m=+0.458063426 container attach 4a49b29c2fc94847b1c19e16a3d01a747b8c13f7816ccb180146cd2d4d9bda65 (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-0-deactivate, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-c2ddf97, ceph=True, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-17T00:32:39.229 INFO:journalctl@ceph.osd.0.smithi002.stdout:Sep 17 00:32:38 smithi002 podman[95231]: 2024-09-17 00:32:38.988795973 +0000 UTC m=+0.554861706 container died 4a49b29c2fc94847b1c19e16a3d01a747b8c13f7816ccb180146cd2d4d9bda65 (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-0-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=squid-c2ddf97, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-09-17T00:32:39.563 DEBUG:teuthology.orchestra.run.smithi002:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3aad27a2-7489-11ef-bceb-c7b262605968@osd.0.service' 2024-09-17T00:32:39.608 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-17T00:32:39.608 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-09-17T00:32:39.608 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-09-17T00:32:39.608 DEBUG:teuthology.orchestra.run.smithi002:> sudo systemctl stop ceph-3aad27a2-7489-11ef-bceb-c7b262605968@osd.1 2024-09-17T00:32:39.828 INFO:journalctl@ceph.osd.1.smithi002.stdout:Sep 17 00:32:39 smithi002 systemd[1]: Stopping Ceph osd.1 for 3aad27a2-7489-11ef-bceb-c7b262605968... 2024-09-17T00:32:40.151 INFO:journalctl@ceph.osd.1.smithi002.stdout:Sep 17 00:32:39 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-1[74140]: 2024-09-17T00:32:39.982+0000 7fd005aee640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-17T00:32:40.152 INFO:journalctl@ceph.osd.1.smithi002.stdout:Sep 17 00:32:39 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-1[74140]: 2024-09-17T00:32:39.982+0000 7fd005aee640 -1 osd.1 153 *** Got signal Terminated *** 2024-09-17T00:32:40.152 INFO:journalctl@ceph.osd.1.smithi002.stdout:Sep 17 00:32:39 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-1[74140]: 2024-09-17T00:32:39.982+0000 7fd005aee640 -1 osd.1 153 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-17T00:32:44.729 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Sep 17 00:32:44 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-alertmanager-a[58000]: ts=2024-09-17T00:32:44.238Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi002.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.2:8443: connect: connection refused" 2024-09-17T00:32:44.729 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Sep 17 00:32:44 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-alertmanager-a[58000]: ts=2024-09-17T00:32:44.239Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi099.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.99:8443: connect: connection refused" 2024-09-17T00:32:45.679 INFO:journalctl@ceph.osd.1.smithi002.stdout:Sep 17 00:32:45 smithi002 podman[95335]: 2024-09-17 00:32:45.392401391 +0000 UTC m=+5.528742905 container died 3dbc23d57c5342b75bf62f260234f2a9b2d29ffb9fd01765e9bb0b5589299b23 (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-1, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=squid-c2ddf97, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-09-17T00:32:45.979 INFO:journalctl@ceph.osd.1.smithi002.stdout:Sep 17 00:32:45 smithi002 podman[95335]: 2024-09-17 00:32:45.773161479 +0000 UTC m=+5.909502980 container cleanup 3dbc23d57c5342b75bf62f260234f2a9b2d29ffb9fd01765e9bb0b5589299b23 (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-1, GIT_CLEAN=True, RELEASE=squid-c2ddf97, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2) 2024-09-17T00:32:45.979 INFO:journalctl@ceph.osd.1.smithi002.stdout:Sep 17 00:32:45 smithi002 bash[95335]: ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-1 2024-09-17T00:32:46.393 INFO:journalctl@ceph.osd.1.smithi002.stdout:Sep 17 00:32:46 smithi002 podman[95362]: 2024-09-17 00:32:46.066302529 +0000 UTC m=+0.672244385 container remove 3dbc23d57c5342b75bf62f260234f2a9b2d29ffb9fd01765e9bb0b5589299b23 (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-1, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=squid-c2ddf97, ceph=True, CEPH_POINT_RELEASE=) 2024-09-17T00:32:46.675 INFO:journalctl@ceph.osd.1.smithi002.stdout:Sep 17 00:32:46 smithi002 podman[95451]: 2024-09-17 00:32:46.299346255 +0000 UTC m=+0.016466472 image pull ebb0615149def4a3179c82c0cc47af75808300abb9f1142788e5e605a7ac0e5b quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f 2024-09-17T00:32:46.675 INFO:journalctl@ceph.osd.1.smithi002.stdout:Sep 17 00:32:46 smithi002 podman[95451]: 2024-09-17 00:32:46.442366782 +0000 UTC m=+0.159486982 container create 8785c218fa13f00093572a683c57a434b826696eef8736633724d59924c85432 (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-1-deactivate, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=squid-c2ddf97) 2024-09-17T00:32:46.675 INFO:journalctl@ceph.osd.1.smithi002.stdout:Sep 17 00:32:46 smithi002 podman[95451]: 2024-09-17 00:32:46.678222459 +0000 UTC m=+0.395342671 container init 8785c218fa13f00093572a683c57a434b826696eef8736633724d59924c85432 (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-1-deactivate, RELEASE=squid-c2ddf97, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-17T00:32:46.979 INFO:journalctl@ceph.osd.1.smithi002.stdout:Sep 17 00:32:46 smithi002 podman[95451]: 2024-09-17 00:32:46.681838679 +0000 UTC m=+0.398958877 container start 8785c218fa13f00093572a683c57a434b826696eef8736633724d59924c85432 (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-1-deactivate, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=squid-c2ddf97) 2024-09-17T00:32:46.979 INFO:journalctl@ceph.osd.1.smithi002.stdout:Sep 17 00:32:46 smithi002 podman[95451]: 2024-09-17 00:32:46.769857098 +0000 UTC m=+0.486977304 container attach 8785c218fa13f00093572a683c57a434b826696eef8736633724d59924c85432 (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-1-deactivate, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=squid-c2ddf97, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD) 2024-09-17T00:32:46.979 INFO:journalctl@ceph.osd.1.smithi002.stdout:Sep 17 00:32:46 smithi002 podman[95451]: 2024-09-17 00:32:46.874053202 +0000 UTC m=+0.591173411 container died 8785c218fa13f00093572a683c57a434b826696eef8736633724d59924c85432 (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-1-deactivate, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, RELEASE=squid-c2ddf97, org.label-schema.license=GPLv2) 2024-09-17T00:32:47.570 DEBUG:teuthology.orchestra.run.smithi002:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3aad27a2-7489-11ef-bceb-c7b262605968@osd.1.service' 2024-09-17T00:32:47.615 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-17T00:32:47.615 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-09-17T00:32:47.615 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-09-17T00:32:47.615 DEBUG:teuthology.orchestra.run.smithi002:> sudo systemctl stop ceph-3aad27a2-7489-11ef-bceb-c7b262605968@osd.2 2024-09-17T00:32:47.949 INFO:journalctl@ceph.osd.2.smithi002.stdout:Sep 17 00:32:47 smithi002 systemd[1]: Stopping Ceph osd.2 for 3aad27a2-7489-11ef-bceb-c7b262605968... 2024-09-17T00:32:48.229 INFO:journalctl@ceph.osd.2.smithi002.stdout:Sep 17 00:32:47 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-2[77190]: 2024-09-17T00:32:47.951+0000 7f307d4cf640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-17T00:32:48.229 INFO:journalctl@ceph.osd.2.smithi002.stdout:Sep 17 00:32:47 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-2[77190]: 2024-09-17T00:32:47.951+0000 7f307d4cf640 -1 osd.2 153 *** Got signal Terminated *** 2024-09-17T00:32:48.229 INFO:journalctl@ceph.osd.2.smithi002.stdout:Sep 17 00:32:47 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-2[77190]: 2024-09-17T00:32:47.951+0000 7f307d4cf640 -1 osd.2 153 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-17T00:32:53.714 INFO:journalctl@ceph.osd.2.smithi002.stdout:Sep 17 00:32:53 smithi002 podman[95555]: 2024-09-17 00:32:53.398600452 +0000 UTC m=+5.549973711 container died d21401b6763b6ee53aa5fd95e387f129c3cfec4c07c6de3fae4bda256a1e9d32 (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=squid-c2ddf97, ceph=True, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-09-17T00:32:53.715 INFO:journalctl@ceph.osd.2.smithi002.stdout:Sep 17 00:32:53 smithi002 podman[95555]: 2024-09-17 00:32:53.717559056 +0000 UTC m=+5.868932305 container cleanup d21401b6763b6ee53aa5fd95e387f129c3cfec4c07c6de3fae4bda256a1e9d32 (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-2, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, RELEASE=squid-c2ddf97, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-09-17T00:32:53.979 INFO:journalctl@ceph.osd.2.smithi002.stdout:Sep 17 00:32:53 smithi002 bash[95555]: ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-2 2024-09-17T00:32:53.979 INFO:journalctl@ceph.osd.2.smithi002.stdout:Sep 17 00:32:53 smithi002 podman[95573]: 2024-09-17 00:32:53.951998806 +0000 UTC m=+0.551572658 container remove d21401b6763b6ee53aa5fd95e387f129c3cfec4c07c6de3fae4bda256a1e9d32 (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-2, RELEASE=squid-c2ddf97, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux ) 2024-09-17T00:32:54.517 INFO:journalctl@ceph.osd.2.smithi002.stdout:Sep 17 00:32:54 smithi002 podman[95661]: 2024-09-17 00:32:54.295421361 +0000 UTC m=+0.016446704 image pull ebb0615149def4a3179c82c0cc47af75808300abb9f1142788e5e605a7ac0e5b quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f 2024-09-17T00:32:54.518 INFO:journalctl@ceph.osd.2.smithi002.stdout:Sep 17 00:32:54 smithi002 podman[95661]: 2024-09-17 00:32:54.411149423 +0000 UTC m=+0.132174754 container create 16b23602a8a22bf189736c31a131fc66a98ea600bbed02e1c5e460f745307993 (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-2-deactivate, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-c2ddf97, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, io.buildah.version=1.37.2) 2024-09-17T00:32:54.518 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Sep 17 00:32:54 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-alertmanager-a[58000]: ts=2024-09-17T00:32:54.238Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi099.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.99:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi002.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.2:8443: connect: connection refused" 2024-09-17T00:32:54.518 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Sep 17 00:32:54 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-alertmanager-a[58000]: ts=2024-09-17T00:32:54.239Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi002.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.2:8443: connect: connection refused" 2024-09-17T00:32:54.518 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Sep 17 00:32:54 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-alertmanager-a[58000]: ts=2024-09-17T00:32:54.239Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi099.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.99:8443: connect: connection refused" 2024-09-17T00:32:54.810 INFO:journalctl@ceph.osd.2.smithi002.stdout:Sep 17 00:32:54 smithi002 podman[95661]: 2024-09-17 00:32:54.613719694 +0000 UTC m=+0.334745028 container init 16b23602a8a22bf189736c31a131fc66a98ea600bbed02e1c5e460f745307993 (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-2-deactivate, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-c2ddf97, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.schema-version=1.0) 2024-09-17T00:32:54.810 INFO:journalctl@ceph.osd.2.smithi002.stdout:Sep 17 00:32:54 smithi002 podman[95661]: 2024-09-17 00:32:54.617176827 +0000 UTC m=+0.338202151 container start 16b23602a8a22bf189736c31a131fc66a98ea600bbed02e1c5e460f745307993 (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-2-deactivate, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-c2ddf97, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0) 2024-09-17T00:32:54.810 INFO:journalctl@ceph.osd.2.smithi002.stdout:Sep 17 00:32:54 smithi002 podman[95661]: 2024-09-17 00:32:54.68853524 +0000 UTC m=+0.409560577 container attach 16b23602a8a22bf189736c31a131fc66a98ea600bbed02e1c5e460f745307993 (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-2-deactivate, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-c2ddf97, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, org.label-schema.license=GPLv2) 2024-09-17T00:32:55.155 INFO:journalctl@ceph.osd.2.smithi002.stdout:Sep 17 00:32:54 smithi002 podman[95661]: 2024-09-17 00:32:54.813833824 +0000 UTC m=+0.534859168 container died 16b23602a8a22bf189736c31a131fc66a98ea600bbed02e1c5e460f745307993 (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-2-deactivate, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, RELEASE=squid-c2ddf97, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-17T00:32:55.547 DEBUG:teuthology.orchestra.run.smithi002:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3aad27a2-7489-11ef-bceb-c7b262605968@osd.2.service' 2024-09-17T00:32:55.596 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-17T00:32:55.596 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-09-17T00:32:55.596 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-09-17T00:32:55.596 DEBUG:teuthology.orchestra.run.smithi002:> sudo systemctl stop ceph-3aad27a2-7489-11ef-bceb-c7b262605968@osd.3 2024-09-17T00:32:55.901 INFO:journalctl@ceph.osd.3.smithi002.stdout:Sep 17 00:32:55 smithi002 systemd[1]: Stopping Ceph osd.3 for 3aad27a2-7489-11ef-bceb-c7b262605968... 2024-09-17T00:32:56.229 INFO:journalctl@ceph.osd.3.smithi002.stdout:Sep 17 00:32:55 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-3[80109]: 2024-09-17T00:32:55.903+0000 7f34af6fe640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-17T00:32:56.229 INFO:journalctl@ceph.osd.3.smithi002.stdout:Sep 17 00:32:55 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-3[80109]: 2024-09-17T00:32:55.903+0000 7f34af6fe640 -1 osd.3 153 *** Got signal Terminated *** 2024-09-17T00:32:56.229 INFO:journalctl@ceph.osd.3.smithi002.stdout:Sep 17 00:32:55 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-3[80109]: 2024-09-17T00:32:55.903+0000 7f34af6fe640 -1 osd.3 153 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-17T00:32:58.967 INFO:journalctl@ceph.osd.5.smithi099.stdout:Sep 17 00:32:58 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-5[62260]: 2024-09-17T00:32:58.585+0000 7f186ef88640 -1 osd.5 153 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-09-17T00:32:36.894841+0000 front 2024-09-17T00:32:36.894795+0000 (oldest deadline 2024-09-17T00:32:57.994681+0000) 2024-09-17T00:32:58.967 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:32:58 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-7[66902]: 2024-09-17T00:32:58.557+0000 7f5a4e5b9640 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-09-17T00:32:33.676078+0000 front 2024-09-17T00:32:33.676062+0000 (oldest deadline 2024-09-17T00:32:57.775877+0000) 2024-09-17T00:32:59.717 INFO:journalctl@ceph.osd.5.smithi099.stdout:Sep 17 00:32:59 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-5[62260]: 2024-09-17T00:32:59.605+0000 7f186ef88640 -1 osd.5 153 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-09-17T00:32:36.894841+0000 front 2024-09-17T00:32:36.894795+0000 (oldest deadline 2024-09-17T00:32:57.994681+0000) 2024-09-17T00:32:59.717 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:32:59 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-7[66902]: 2024-09-17T00:32:59.534+0000 7f5a4e5b9640 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-09-17T00:32:33.676078+0000 front 2024-09-17T00:32:33.676062+0000 (oldest deadline 2024-09-17T00:32:57.775877+0000) 2024-09-17T00:32:59.979 INFO:journalctl@ceph.osd.3.smithi002.stdout:Sep 17 00:32:59 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-3[80109]: 2024-09-17T00:32:59.548+0000 7f34abd14640 -1 osd.3 153 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-09-17T00:32:36.832292+0000 front 2024-09-17T00:32:36.832147+0000 (oldest deadline 2024-09-17T00:32:59.132332+0000) 2024-09-17T00:33:00.151 INFO:journalctl@ceph.osd.6.smithi099.stdout:Sep 17 00:32:59 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-6[64592]: 2024-09-17T00:32:59.758+0000 7f839b3b6640 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-09-17T00:32:34.544008+0000 front 2024-09-17T00:32:34.544050+0000 (oldest deadline 2024-09-17T00:32:59.243805+0000) 2024-09-17T00:33:00.930 INFO:journalctl@ceph.osd.6.smithi099.stdout:Sep 17 00:33:00 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-6[64592]: 2024-09-17T00:33:00.715+0000 7f839b3b6640 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-09-17T00:32:34.544008+0000 front 2024-09-17T00:32:34.544050+0000 (oldest deadline 2024-09-17T00:32:59.243805+0000) 2024-09-17T00:33:00.931 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:33:00 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-7[66902]: 2024-09-17T00:33:00.491+0000 7f5a4e5b9640 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-09-17T00:32:33.676078+0000 front 2024-09-17T00:32:33.676062+0000 (oldest deadline 2024-09-17T00:32:57.775877+0000) 2024-09-17T00:33:00.931 INFO:journalctl@ceph.osd.5.smithi099.stdout:Sep 17 00:33:00 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-5[62260]: 2024-09-17T00:33:00.644+0000 7f186ef88640 -1 osd.5 153 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-09-17T00:32:36.894841+0000 front 2024-09-17T00:32:36.894795+0000 (oldest deadline 2024-09-17T00:32:57.994681+0000) 2024-09-17T00:33:00.979 INFO:journalctl@ceph.osd.3.smithi002.stdout:Sep 17 00:33:00 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-3[80109]: 2024-09-17T00:33:00.565+0000 7f34abd14640 -1 osd.3 153 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-09-17T00:32:36.832292+0000 front 2024-09-17T00:32:36.832147+0000 (oldest deadline 2024-09-17T00:32:59.132332+0000) 2024-09-17T00:33:01.217 INFO:journalctl@ceph.osd.4.smithi099.stdout:Sep 17 00:33:00 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-4[59931]: 2024-09-17T00:33:00.933+0000 7f462015b640 -1 osd.4 153 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-09-17T00:32:34.955381+0000 front 2024-09-17T00:32:34.955309+0000 (oldest deadline 2024-09-17T00:33:00.255219+0000) 2024-09-17T00:33:01.561 INFO:journalctl@ceph.osd.3.smithi002.stdout:Sep 17 00:33:01 smithi002 podman[95763]: 2024-09-17 00:33:01.283231371 +0000 UTC m=+5.485997916 container died 980f2a25d096db44b54da22ac4e7c1587a0533a4815afd6f1769a6f3ea3f1a22 (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-3, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, RELEASE=squid-c2ddf97, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-09-17T00:33:01.961 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:33:01 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-7[66902]: 2024-09-17T00:33:01.513+0000 7f5a4e5b9640 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-09-17T00:32:33.676078+0000 front 2024-09-17T00:32:33.676062+0000 (oldest deadline 2024-09-17T00:32:57.775877+0000) 2024-09-17T00:33:01.962 INFO:journalctl@ceph.osd.5.smithi099.stdout:Sep 17 00:33:01 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-5[62260]: 2024-09-17T00:33:01.624+0000 7f186ef88640 -1 osd.5 153 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-09-17T00:32:36.894841+0000 front 2024-09-17T00:32:36.894795+0000 (oldest deadline 2024-09-17T00:32:57.994681+0000) 2024-09-17T00:33:01.962 INFO:journalctl@ceph.osd.6.smithi099.stdout:Sep 17 00:33:01 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-6[64592]: 2024-09-17T00:33:01.687+0000 7f839b3b6640 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-09-17T00:32:34.544008+0000 front 2024-09-17T00:32:34.544050+0000 (oldest deadline 2024-09-17T00:32:59.243805+0000) 2024-09-17T00:33:01.979 INFO:journalctl@ceph.osd.3.smithi002.stdout:Sep 17 00:33:01 smithi002 podman[95763]: 2024-09-17 00:33:01.745321067 +0000 UTC m=+5.948087654 container cleanup 980f2a25d096db44b54da22ac4e7c1587a0533a4815afd6f1769a6f3ea3f1a22 (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-3, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, RELEASE=squid-c2ddf97, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-17T00:33:01.979 INFO:journalctl@ceph.osd.3.smithi002.stdout:Sep 17 00:33:01 smithi002 bash[95763]: ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-3 2024-09-17T00:33:02.217 INFO:journalctl@ceph.osd.4.smithi099.stdout:Sep 17 00:33:01 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-4[59931]: 2024-09-17T00:33:01.964+0000 7f462015b640 -1 osd.4 153 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-09-17T00:32:34.955381+0000 front 2024-09-17T00:32:34.955309+0000 (oldest deadline 2024-09-17T00:33:00.255219+0000) 2024-09-17T00:33:02.373 INFO:journalctl@ceph.osd.3.smithi002.stdout:Sep 17 00:33:02 smithi002 podman[95779]: 2024-09-17 00:33:02.029733091 +0000 UTC m=+0.744367150 container remove 980f2a25d096db44b54da22ac4e7c1587a0533a4815afd6f1769a6f3ea3f1a22 (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-3, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240731, RELEASE=squid-c2ddf97, CEPH_POINT_RELEASE=) 2024-09-17T00:33:02.625 INFO:journalctl@ceph.osd.3.smithi002.stdout:Sep 17 00:33:02 smithi002 podman[95871]: 2024-09-17 00:33:02.27972531 +0000 UTC m=+0.016393747 image pull ebb0615149def4a3179c82c0cc47af75808300abb9f1142788e5e605a7ac0e5b quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f 2024-09-17T00:33:02.625 INFO:journalctl@ceph.osd.3.smithi002.stdout:Sep 17 00:33:02 smithi002 podman[95871]: 2024-09-17 00:33:02.389171541 +0000 UTC m=+0.125839969 container create 77181bea758a56d8985500935fc6dd7956de16b3d130b64b3a04626eac72a5de (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-3-deactivate, RELEASE=squid-c2ddf97, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-09-17T00:33:02.625 INFO:journalctl@ceph.osd.3.smithi002.stdout:Sep 17 00:33:02 smithi002 podman[95871]: 2024-09-17 00:33:02.625011262 +0000 UTC m=+0.361679695 container init 77181bea758a56d8985500935fc6dd7956de16b3d130b64b3a04626eac72a5de (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-3-deactivate, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=squid-c2ddf97) 2024-09-17T00:33:02.625 INFO:journalctl@ceph.osd.3.smithi002.stdout:Sep 17 00:33:02 smithi002 podman[95871]: 2024-09-17 00:33:02.628543532 +0000 UTC m=+0.365211964 container start 77181bea758a56d8985500935fc6dd7956de16b3d130b64b3a04626eac72a5de (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-3-deactivate, RELEASE=squid-c2ddf97, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_CLEAN=True, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS) 2024-09-17T00:33:02.967 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:33:02 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-7[66902]: 2024-09-17T00:33:02.499+0000 7f5a4e5b9640 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-09-17T00:32:33.676078+0000 front 2024-09-17T00:32:33.676062+0000 (oldest deadline 2024-09-17T00:32:57.775877+0000) 2024-09-17T00:33:02.967 INFO:journalctl@ceph.osd.5.smithi099.stdout:Sep 17 00:33:02 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-5[62260]: 2024-09-17T00:33:02.664+0000 7f186ef88640 -1 osd.5 153 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-09-17T00:32:36.894841+0000 front 2024-09-17T00:32:36.894795+0000 (oldest deadline 2024-09-17T00:32:57.994681+0000) 2024-09-17T00:33:02.968 INFO:journalctl@ceph.osd.6.smithi099.stdout:Sep 17 00:33:02 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-6[64592]: 2024-09-17T00:33:02.674+0000 7f839b3b6640 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-09-17T00:32:34.544008+0000 front 2024-09-17T00:32:34.544050+0000 (oldest deadline 2024-09-17T00:32:59.243805+0000) 2024-09-17T00:33:02.979 INFO:journalctl@ceph.osd.3.smithi002.stdout:Sep 17 00:33:02 smithi002 podman[95871]: 2024-09-17 00:33:02.733513512 +0000 UTC m=+0.470181951 container attach 77181bea758a56d8985500935fc6dd7956de16b3d130b64b3a04626eac72a5de (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-3-deactivate, RELEASE=squid-c2ddf97, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True) 2024-09-17T00:33:02.979 INFO:journalctl@ceph.osd.3.smithi002.stdout:Sep 17 00:33:02 smithi002 podman[95871]: 2024-09-17 00:33:02.82783178 +0000 UTC m=+0.564500210 container died 77181bea758a56d8985500935fc6dd7956de16b3d130b64b3a04626eac72a5de (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-3-deactivate, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, RELEASE=squid-c2ddf97) 2024-09-17T00:33:03.450 INFO:journalctl@ceph.osd.4.smithi099.stdout:Sep 17 00:33:03 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-4[59931]: 2024-09-17T00:33:03.005+0000 7f462015b640 -1 osd.4 153 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-09-17T00:32:34.955381+0000 front 2024-09-17T00:32:34.955309+0000 (oldest deadline 2024-09-17T00:33:00.255219+0000) 2024-09-17T00:33:03.458 DEBUG:teuthology.orchestra.run.smithi002:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3aad27a2-7489-11ef-bceb-c7b262605968@osd.3.service' 2024-09-17T00:33:03.498 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-17T00:33:03.499 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-09-17T00:33:03.499 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-09-17T00:33:03.499 DEBUG:teuthology.orchestra.run.smithi099:> sudo systemctl stop ceph-3aad27a2-7489-11ef-bceb-c7b262605968@osd.4 2024-09-17T00:33:03.717 INFO:journalctl@ceph.osd.4.smithi099.stdout:Sep 17 00:33:03 smithi099 systemd[1]: Stopping Ceph osd.4 for 3aad27a2-7489-11ef-bceb-c7b262605968... 2024-09-17T00:33:03.717 INFO:journalctl@ceph.osd.6.smithi099.stdout:Sep 17 00:33:03 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-6[64592]: 2024-09-17T00:33:03.694+0000 7f839b3b6640 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-09-17T00:32:34.544008+0000 front 2024-09-17T00:32:34.544050+0000 (oldest deadline 2024-09-17T00:32:59.243805+0000) 2024-09-17T00:33:03.717 INFO:journalctl@ceph.osd.5.smithi099.stdout:Sep 17 00:33:03 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-5[62260]: 2024-09-17T00:33:03.647+0000 7f186ef88640 -1 osd.5 153 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-09-17T00:32:36.894841+0000 front 2024-09-17T00:32:36.894795+0000 (oldest deadline 2024-09-17T00:32:57.994681+0000) 2024-09-17T00:33:03.718 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:33:03 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-7[66902]: 2024-09-17T00:33:03.452+0000 7f5a4e5b9640 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-09-17T00:32:33.676078+0000 front 2024-09-17T00:32:33.676062+0000 (oldest deadline 2024-09-17T00:32:57.775877+0000) 2024-09-17T00:33:04.217 INFO:journalctl@ceph.osd.4.smithi099.stdout:Sep 17 00:33:03 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-4[59931]: 2024-09-17T00:33:03.892+0000 7f4623b45640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-17T00:33:04.217 INFO:journalctl@ceph.osd.4.smithi099.stdout:Sep 17 00:33:03 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-4[59931]: 2024-09-17T00:33:03.892+0000 7f4623b45640 -1 osd.4 153 *** Got signal Terminated *** 2024-09-17T00:33:04.217 INFO:journalctl@ceph.osd.4.smithi099.stdout:Sep 17 00:33:03 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-4[59931]: 2024-09-17T00:33:03.892+0000 7f4623b45640 -1 osd.4 153 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-17T00:33:04.217 INFO:journalctl@ceph.osd.4.smithi099.stdout:Sep 17 00:33:03 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-4[59931]: 2024-09-17T00:33:03.957+0000 7f462015b640 -1 osd.4 153 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-09-17T00:32:34.955381+0000 front 2024-09-17T00:32:34.955309+0000 (oldest deadline 2024-09-17T00:33:00.255219+0000) 2024-09-17T00:33:04.704 INFO:journalctl@ceph.osd.5.smithi099.stdout:Sep 17 00:33:04 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-5[62260]: 2024-09-17T00:33:04.649+0000 7f186ef88640 -1 osd.5 153 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-09-17T00:32:36.894841+0000 front 2024-09-17T00:32:36.894795+0000 (oldest deadline 2024-09-17T00:32:57.994681+0000) 2024-09-17T00:33:04.704 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:33:04 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-7[66902]: 2024-09-17T00:33:04.428+0000 7f5a4e5b9640 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-09-17T00:32:33.676078+0000 front 2024-09-17T00:32:33.676062+0000 (oldest deadline 2024-09-17T00:32:57.775877+0000) 2024-09-17T00:33:04.729 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Sep 17 00:33:04 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-alertmanager-a[58000]: ts=2024-09-17T00:33:04.239Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi099.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.99:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi002.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.2:8443: connect: connection refused" 2024-09-17T00:33:04.729 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Sep 17 00:33:04 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-alertmanager-a[58000]: ts=2024-09-17T00:33:04.239Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi002.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.2:8443: connect: connection refused" 2024-09-17T00:33:04.729 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Sep 17 00:33:04 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-alertmanager-a[58000]: ts=2024-09-17T00:33:04.240Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi099.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.99:8443: connect: connection refused" 2024-09-17T00:33:04.967 INFO:journalctl@ceph.osd.6.smithi099.stdout:Sep 17 00:33:04 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-6[64592]: 2024-09-17T00:33:04.706+0000 7f839b3b6640 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-09-17T00:32:34.544008+0000 front 2024-09-17T00:32:34.544050+0000 (oldest deadline 2024-09-17T00:32:59.243805+0000) 2024-09-17T00:33:04.967 INFO:journalctl@ceph.osd.4.smithi099.stdout:Sep 17 00:33:04 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-4[59931]: 2024-09-17T00:33:04.949+0000 7f462015b640 -1 osd.4 153 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-09-17T00:32:34.955381+0000 front 2024-09-17T00:32:34.955309+0000 (oldest deadline 2024-09-17T00:33:00.255219+0000) 2024-09-17T00:33:05.716 INFO:journalctl@ceph.osd.5.smithi099.stdout:Sep 17 00:33:05 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-5[62260]: 2024-09-17T00:33:05.672+0000 7f186ef88640 -1 osd.5 153 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-09-17T00:32:36.894841+0000 front 2024-09-17T00:32:36.894795+0000 (oldest deadline 2024-09-17T00:32:57.994681+0000) 2024-09-17T00:33:05.716 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:33:05 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-7[66902]: 2024-09-17T00:33:05.432+0000 7f5a4e5b9640 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-09-17T00:32:33.676078+0000 front 2024-09-17T00:32:33.676062+0000 (oldest deadline 2024-09-17T00:32:57.775877+0000) 2024-09-17T00:33:05.967 INFO:journalctl@ceph.osd.4.smithi099.stdout:Sep 17 00:33:05 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-4[59931]: 2024-09-17T00:33:05.965+0000 7f462015b640 -1 osd.4 153 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-09-17T00:32:34.955381+0000 front 2024-09-17T00:32:34.955309+0000 (oldest deadline 2024-09-17T00:33:00.255219+0000) 2024-09-17T00:33:05.967 INFO:journalctl@ceph.osd.6.smithi099.stdout:Sep 17 00:33:05 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-6[64592]: 2024-09-17T00:33:05.719+0000 7f839b3b6640 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-09-17T00:32:34.544008+0000 front 2024-09-17T00:32:34.544050+0000 (oldest deadline 2024-09-17T00:32:59.243805+0000) 2024-09-17T00:33:06.690 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:33:06 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-7[66902]: 2024-09-17T00:33:06.396+0000 7f5a4e5b9640 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-09-17T00:32:33.676078+0000 front 2024-09-17T00:32:33.676062+0000 (oldest deadline 2024-09-17T00:32:57.775877+0000) 2024-09-17T00:33:06.941 INFO:journalctl@ceph.osd.5.smithi099.stdout:Sep 17 00:33:06 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-5[62260]: 2024-09-17T00:33:06.693+0000 7f186ef88640 -1 osd.5 153 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-09-17T00:32:36.894841+0000 front 2024-09-17T00:32:36.894795+0000 (oldest deadline 2024-09-17T00:32:57.994681+0000) 2024-09-17T00:33:06.941 INFO:journalctl@ceph.osd.6.smithi099.stdout:Sep 17 00:33:06 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-6[64592]: 2024-09-17T00:33:06.736+0000 7f839b3b6640 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-09-17T00:32:34.544008+0000 front 2024-09-17T00:32:34.544050+0000 (oldest deadline 2024-09-17T00:32:59.243805+0000) 2024-09-17T00:33:07.217 INFO:journalctl@ceph.osd.4.smithi099.stdout:Sep 17 00:33:06 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-4[59931]: 2024-09-17T00:33:06.943+0000 7f462015b640 -1 osd.4 153 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-09-17T00:32:34.955381+0000 front 2024-09-17T00:32:34.955309+0000 (oldest deadline 2024-09-17T00:33:00.255219+0000) 2024-09-17T00:33:07.694 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:33:07 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-7[66902]: 2024-09-17T00:33:07.424+0000 7f5a4e5b9640 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-09-17T00:32:33.676078+0000 front 2024-09-17T00:32:33.676062+0000 (oldest deadline 2024-09-17T00:32:57.775877+0000) 2024-09-17T00:33:07.954 INFO:journalctl@ceph.osd.6.smithi099.stdout:Sep 17 00:33:07 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-6[64592]: 2024-09-17T00:33:07.777+0000 7f839b3b6640 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-09-17T00:32:34.544008+0000 front 2024-09-17T00:32:34.544050+0000 (oldest deadline 2024-09-17T00:32:59.243805+0000) 2024-09-17T00:33:07.955 INFO:journalctl@ceph.osd.5.smithi099.stdout:Sep 17 00:33:07 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-5[62260]: 2024-09-17T00:33:07.696+0000 7f186ef88640 -1 osd.5 153 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-09-17T00:32:36.894841+0000 front 2024-09-17T00:32:36.894795+0000 (oldest deadline 2024-09-17T00:32:57.994681+0000) 2024-09-17T00:33:07.955 INFO:journalctl@ceph.osd.5.smithi099.stdout:Sep 17 00:33:07 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-5[62260]: 2024-09-17T00:33:07.696+0000 7f186ef88640 -1 osd.5 153 heartbeat_check: no reply from 172.21.15.2:6814 osd.1 since back 2024-09-17T00:32:43.895924+0000 front 2024-09-17T00:32:43.895819+0000 (oldest deadline 2024-09-17T00:33:07.395702+0000) 2024-09-17T00:33:08.217 INFO:journalctl@ceph.osd.4.smithi099.stdout:Sep 17 00:33:07 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-4[59931]: 2024-09-17T00:33:07.957+0000 7f462015b640 -1 osd.4 153 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-09-17T00:32:34.955381+0000 front 2024-09-17T00:32:34.955309+0000 (oldest deadline 2024-09-17T00:33:00.255219+0000) 2024-09-17T00:33:08.697 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:33:08 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-7[66902]: 2024-09-17T00:33:08.377+0000 7f5a4e5b9640 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-09-17T00:32:33.676078+0000 front 2024-09-17T00:32:33.676062+0000 (oldest deadline 2024-09-17T00:32:57.775877+0000) 2024-09-17T00:33:08.697 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:33:08 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-7[66902]: 2024-09-17T00:33:08.377+0000 7f5a4e5b9640 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.2:6814 osd.1 since back 2024-09-17T00:32:44.777292+0000 front 2024-09-17T00:32:44.777224+0000 (oldest deadline 2024-09-17T00:33:08.277122+0000) 2024-09-17T00:33:08.967 INFO:journalctl@ceph.osd.6.smithi099.stdout:Sep 17 00:33:08 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-6[64592]: 2024-09-17T00:33:08.810+0000 7f839b3b6640 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-09-17T00:32:34.544008+0000 front 2024-09-17T00:32:34.544050+0000 (oldest deadline 2024-09-17T00:32:59.243805+0000) 2024-09-17T00:33:08.967 INFO:journalctl@ceph.osd.5.smithi099.stdout:Sep 17 00:33:08 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-5[62260]: 2024-09-17T00:33:08.699+0000 7f186ef88640 -1 osd.5 153 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-09-17T00:32:36.894841+0000 front 2024-09-17T00:32:36.894795+0000 (oldest deadline 2024-09-17T00:32:57.994681+0000) 2024-09-17T00:33:08.967 INFO:journalctl@ceph.osd.5.smithi099.stdout:Sep 17 00:33:08 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-5[62260]: 2024-09-17T00:33:08.699+0000 7f186ef88640 -1 osd.5 153 heartbeat_check: no reply from 172.21.15.2:6814 osd.1 since back 2024-09-17T00:32:43.895924+0000 front 2024-09-17T00:32:43.895819+0000 (oldest deadline 2024-09-17T00:33:07.395702+0000) 2024-09-17T00:33:09.649 INFO:journalctl@ceph.osd.4.smithi099.stdout:Sep 17 00:33:09 smithi099 podman[75381]: 2024-09-17 00:33:09.382566449 +0000 UTC m=+5.681212406 container died f644e5be4db8ad107c096c308c1c5c59a9750602e759a4fdb7b95da1b60a89ec (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-4, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=squid-c2ddf97, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True) 2024-09-17T00:33:09.649 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:33:09 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-7[66902]: 2024-09-17T00:33:09.348+0000 7f5a4e5b9640 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-09-17T00:32:33.676078+0000 front 2024-09-17T00:32:33.676062+0000 (oldest deadline 2024-09-17T00:32:57.775877+0000) 2024-09-17T00:33:09.649 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:33:09 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-7[66902]: 2024-09-17T00:33:09.348+0000 7f5a4e5b9640 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.2:6814 osd.1 since back 2024-09-17T00:32:44.777292+0000 front 2024-09-17T00:32:44.777224+0000 (oldest deadline 2024-09-17T00:33:08.277122+0000) 2024-09-17T00:33:09.967 INFO:journalctl@ceph.osd.6.smithi099.stdout:Sep 17 00:33:09 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-6[64592]: 2024-09-17T00:33:09.785+0000 7f839b3b6640 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-09-17T00:32:34.544008+0000 front 2024-09-17T00:32:34.544050+0000 (oldest deadline 2024-09-17T00:32:59.243805+0000) 2024-09-17T00:33:09.967 INFO:journalctl@ceph.osd.5.smithi099.stdout:Sep 17 00:33:09 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-5[62260]: 2024-09-17T00:33:09.728+0000 7f186ef88640 -1 osd.5 153 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-09-17T00:32:36.894841+0000 front 2024-09-17T00:32:36.894795+0000 (oldest deadline 2024-09-17T00:32:57.994681+0000) 2024-09-17T00:33:09.967 INFO:journalctl@ceph.osd.5.smithi099.stdout:Sep 17 00:33:09 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-5[62260]: 2024-09-17T00:33:09.728+0000 7f186ef88640 -1 osd.5 153 heartbeat_check: no reply from 172.21.15.2:6814 osd.1 since back 2024-09-17T00:32:43.895924+0000 front 2024-09-17T00:32:43.895819+0000 (oldest deadline 2024-09-17T00:33:07.395702+0000) 2024-09-17T00:33:10.467 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:33:10 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-7[66902]: 2024-09-17T00:33:10.306+0000 7f5a4e5b9640 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-09-17T00:32:33.676078+0000 front 2024-09-17T00:32:33.676062+0000 (oldest deadline 2024-09-17T00:32:57.775877+0000) 2024-09-17T00:33:10.467 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:33:10 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-7[66902]: 2024-09-17T00:33:10.306+0000 7f5a4e5b9640 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.2:6814 osd.1 since back 2024-09-17T00:32:44.777292+0000 front 2024-09-17T00:32:44.777224+0000 (oldest deadline 2024-09-17T00:33:08.277122+0000) 2024-09-17T00:33:10.467 INFO:journalctl@ceph.osd.4.smithi099.stdout:Sep 17 00:33:10 smithi099 podman[75381]: 2024-09-17 00:33:10.080632601 +0000 UTC m=+6.379278547 container cleanup f644e5be4db8ad107c096c308c1c5c59a9750602e759a4fdb7b95da1b60a89ec (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-4, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, RELEASE=squid-c2ddf97, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS) 2024-09-17T00:33:10.467 INFO:journalctl@ceph.osd.4.smithi099.stdout:Sep 17 00:33:10 smithi099 bash[75381]: ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-4 2024-09-17T00:33:10.468 INFO:journalctl@ceph.osd.4.smithi099.stdout:Sep 17 00:33:10 smithi099 podman[75396]: 2024-09-17 00:33:10.2778754 +0000 UTC m=+0.889329391 container remove f644e5be4db8ad107c096c308c1c5c59a9750602e759a4fdb7b95da1b60a89ec (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-4, org.label-schema.vendor=CentOS, RELEASE=squid-c2ddf97, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-17T00:33:10.961 INFO:journalctl@ceph.osd.5.smithi099.stdout:Sep 17 00:33:10 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-5[62260]: 2024-09-17T00:33:10.722+0000 7f186ef88640 -1 osd.5 153 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-09-17T00:32:36.894841+0000 front 2024-09-17T00:32:36.894795+0000 (oldest deadline 2024-09-17T00:32:57.994681+0000) 2024-09-17T00:33:10.962 INFO:journalctl@ceph.osd.5.smithi099.stdout:Sep 17 00:33:10 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-5[62260]: 2024-09-17T00:33:10.722+0000 7f186ef88640 -1 osd.5 153 heartbeat_check: no reply from 172.21.15.2:6814 osd.1 since back 2024-09-17T00:32:43.895924+0000 front 2024-09-17T00:32:43.895819+0000 (oldest deadline 2024-09-17T00:33:07.395702+0000) 2024-09-17T00:33:10.962 INFO:journalctl@ceph.osd.6.smithi099.stdout:Sep 17 00:33:10 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-6[64592]: 2024-09-17T00:33:10.793+0000 7f839b3b6640 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-09-17T00:32:34.544008+0000 front 2024-09-17T00:32:34.544050+0000 (oldest deadline 2024-09-17T00:32:59.243805+0000) 2024-09-17T00:33:10.962 INFO:journalctl@ceph.osd.6.smithi099.stdout:Sep 17 00:33:10 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-6[64592]: 2024-09-17T00:33:10.793+0000 7f839b3b6640 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.2:6814 osd.1 since back 2024-09-17T00:32:39.244223+0000 front 2024-09-17T00:32:39.244308+0000 (oldest deadline 2024-09-17T00:33:09.844982+0000) 2024-09-17T00:33:10.962 INFO:journalctl@ceph.osd.4.smithi099.stdout:Sep 17 00:33:10 smithi099 podman[75484]: 2024-09-17 00:33:10.594438151 +0000 UTC m=+0.015928921 image pull ebb0615149def4a3179c82c0cc47af75808300abb9f1142788e5e605a7ac0e5b quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f 2024-09-17T00:33:10.962 INFO:journalctl@ceph.osd.4.smithi099.stdout:Sep 17 00:33:10 smithi099 podman[75484]: 2024-09-17 00:33:10.695546761 +0000 UTC m=+0.117037531 container create 8f8638ba587cbb8978e493cf9ad3e46ac5eb28e992bf72edf860a691b5b38ab7 (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-4-deactivate, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=squid-c2ddf97, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux ) 2024-09-17T00:33:10.963 INFO:journalctl@ceph.osd.4.smithi099.stdout:Sep 17 00:33:10 smithi099 podman[75484]: 2024-09-17 00:33:10.890060242 +0000 UTC m=+0.311551020 container init 8f8638ba587cbb8978e493cf9ad3e46ac5eb28e992bf72edf860a691b5b38ab7 (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-4-deactivate, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, RELEASE=squid-c2ddf97, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-17T00:33:10.963 INFO:journalctl@ceph.osd.4.smithi099.stdout:Sep 17 00:33:10 smithi099 podman[75484]: 2024-09-17 00:33:10.893441948 +0000 UTC m=+0.314932722 container start 8f8638ba587cbb8978e493cf9ad3e46ac5eb28e992bf72edf860a691b5b38ab7 (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-4-deactivate, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, RELEASE=squid-c2ddf97, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-17T00:33:10.963 INFO:journalctl@ceph.osd.4.smithi099.stdout:Sep 17 00:33:10 smithi099 podman[75484]: 2024-09-17 00:33:10.964829042 +0000 UTC m=+0.386319829 container attach 8f8638ba587cbb8978e493cf9ad3e46ac5eb28e992bf72edf860a691b5b38ab7 (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-4-deactivate, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.37.2, RELEASE=squid-c2ddf97, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-17T00:33:11.217 INFO:journalctl@ceph.osd.4.smithi099.stdout:Sep 17 00:33:11 smithi099 podman[75484]: 2024-09-17 00:33:11.081260842 +0000 UTC m=+0.502751613 container died 8f8638ba587cbb8978e493cf9ad3e46ac5eb28e992bf72edf860a691b5b38ab7 (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-4-deactivate, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=squid-c2ddf97, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965) 2024-09-17T00:33:11.665 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:33:11 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-7[66902]: 2024-09-17T00:33:11.280+0000 7f5a4e5b9640 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-09-17T00:32:33.676078+0000 front 2024-09-17T00:32:33.676062+0000 (oldest deadline 2024-09-17T00:32:57.775877+0000) 2024-09-17T00:33:11.665 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:33:11 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-7[66902]: 2024-09-17T00:33:11.280+0000 7f5a4e5b9640 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.2:6814 osd.1 since back 2024-09-17T00:32:44.777292+0000 front 2024-09-17T00:32:44.777224+0000 (oldest deadline 2024-09-17T00:33:08.277122+0000) 2024-09-17T00:33:11.687 DEBUG:teuthology.orchestra.run.smithi099:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3aad27a2-7489-11ef-bceb-c7b262605968@osd.4.service' 2024-09-17T00:33:11.725 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-17T00:33:11.726 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-09-17T00:33:11.726 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-09-17T00:33:11.726 DEBUG:teuthology.orchestra.run.smithi099:> sudo systemctl stop ceph-3aad27a2-7489-11ef-bceb-c7b262605968@osd.5 2024-09-17T00:33:11.967 INFO:journalctl@ceph.osd.5.smithi099.stdout:Sep 17 00:33:11 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-5[62260]: 2024-09-17T00:33:11.728+0000 7f186ef88640 -1 osd.5 153 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-09-17T00:32:36.894841+0000 front 2024-09-17T00:32:36.894795+0000 (oldest deadline 2024-09-17T00:32:57.994681+0000) 2024-09-17T00:33:11.967 INFO:journalctl@ceph.osd.5.smithi099.stdout:Sep 17 00:33:11 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-5[62260]: 2024-09-17T00:33:11.728+0000 7f186ef88640 -1 osd.5 153 heartbeat_check: no reply from 172.21.15.2:6814 osd.1 since back 2024-09-17T00:32:43.895924+0000 front 2024-09-17T00:32:43.895819+0000 (oldest deadline 2024-09-17T00:33:07.395702+0000) 2024-09-17T00:33:11.967 INFO:journalctl@ceph.osd.5.smithi099.stdout:Sep 17 00:33:11 smithi099 systemd[1]: Stopping Ceph osd.5 for 3aad27a2-7489-11ef-bceb-c7b262605968... 2024-09-17T00:33:11.967 INFO:journalctl@ceph.osd.6.smithi099.stdout:Sep 17 00:33:11 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-6[64592]: 2024-09-17T00:33:11.814+0000 7f839b3b6640 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-09-17T00:32:34.544008+0000 front 2024-09-17T00:32:34.544050+0000 (oldest deadline 2024-09-17T00:32:59.243805+0000) 2024-09-17T00:33:11.967 INFO:journalctl@ceph.osd.6.smithi099.stdout:Sep 17 00:33:11 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-6[64592]: 2024-09-17T00:33:11.814+0000 7f839b3b6640 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.2:6814 osd.1 since back 2024-09-17T00:32:39.244223+0000 front 2024-09-17T00:32:39.244308+0000 (oldest deadline 2024-09-17T00:33:09.844982+0000) 2024-09-17T00:33:12.467 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:33:12 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-7[66902]: 2024-09-17T00:33:12.271+0000 7f5a4e5b9640 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-09-17T00:32:33.676078+0000 front 2024-09-17T00:32:33.676062+0000 (oldest deadline 2024-09-17T00:32:57.775877+0000) 2024-09-17T00:33:12.467 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:33:12 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-7[66902]: 2024-09-17T00:33:12.271+0000 7f5a4e5b9640 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.2:6814 osd.1 since back 2024-09-17T00:32:44.777292+0000 front 2024-09-17T00:32:44.777224+0000 (oldest deadline 2024-09-17T00:33:08.277122+0000) 2024-09-17T00:33:12.468 INFO:journalctl@ceph.osd.5.smithi099.stdout:Sep 17 00:33:12 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-5[62260]: 2024-09-17T00:33:12.080+0000 7f187697a640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-17T00:33:12.468 INFO:journalctl@ceph.osd.5.smithi099.stdout:Sep 17 00:33:12 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-5[62260]: 2024-09-17T00:33:12.080+0000 7f187697a640 -1 osd.5 153 *** Got signal Terminated *** 2024-09-17T00:33:12.468 INFO:journalctl@ceph.osd.5.smithi099.stdout:Sep 17 00:33:12 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-5[62260]: 2024-09-17T00:33:12.080+0000 7f187697a640 -1 osd.5 153 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-17T00:33:12.468 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:33:12 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[52757]: ts=2024-09-17T00:33:12.099Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph msg="Unable to refresh target groups" err="Get \"http://172.21.15.2:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.2:8765: connect: connection refused" 2024-09-17T00:33:12.468 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:33:12 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[52757]: ts=2024-09-17T00:33:12.101Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph-exporter msg="Unable to refresh target groups" err="Get \"http://172.21.15.2:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.2:8765: connect: connection refused" 2024-09-17T00:33:12.468 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:33:12 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[52757]: ts=2024-09-17T00:33:12.101Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=nvmeof msg="Unable to refresh target groups" err="Get \"http://172.21.15.2:8765/sd/prometheus/sd-config?service=nvmeof\": dial tcp 172.21.15.2:8765: connect: connection refused" 2024-09-17T00:33:12.469 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:33:12 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[52757]: ts=2024-09-17T00:33:12.101Z caller=refresh.go:99 level=error component="discovery manager notify" discovery=http config=config-0 msg="Unable to refresh target groups" err="Get \"http://172.21.15.2:8765/sd/prometheus/sd-config?service=alertmanager\": dial tcp 172.21.15.2:8765: connect: connection refused" 2024-09-17T00:33:12.469 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:33:12 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[52757]: ts=2024-09-17T00:33:12.101Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=node msg="Unable to refresh target groups" err="Get \"http://172.21.15.2:8765/sd/prometheus/sd-config?service=node-exporter\": dial tcp 172.21.15.2:8765: connect: connection refused" 2024-09-17T00:33:13.217 INFO:journalctl@ceph.osd.6.smithi099.stdout:Sep 17 00:33:12 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-6[64592]: 2024-09-17T00:33:12.774+0000 7f839b3b6640 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-09-17T00:32:34.544008+0000 front 2024-09-17T00:32:34.544050+0000 (oldest deadline 2024-09-17T00:32:59.243805+0000) 2024-09-17T00:33:13.217 INFO:journalctl@ceph.osd.6.smithi099.stdout:Sep 17 00:33:12 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-6[64592]: 2024-09-17T00:33:12.774+0000 7f839b3b6640 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.2:6814 osd.1 since back 2024-09-17T00:32:39.244223+0000 front 2024-09-17T00:32:39.244308+0000 (oldest deadline 2024-09-17T00:33:09.844982+0000) 2024-09-17T00:33:13.217 INFO:journalctl@ceph.osd.5.smithi099.stdout:Sep 17 00:33:12 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-5[62260]: 2024-09-17T00:33:12.775+0000 7f186ef88640 -1 osd.5 153 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-09-17T00:32:36.894841+0000 front 2024-09-17T00:32:36.894795+0000 (oldest deadline 2024-09-17T00:32:57.994681+0000) 2024-09-17T00:33:13.218 INFO:journalctl@ceph.osd.5.smithi099.stdout:Sep 17 00:33:12 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-5[62260]: 2024-09-17T00:33:12.775+0000 7f186ef88640 -1 osd.5 153 heartbeat_check: no reply from 172.21.15.2:6814 osd.1 since back 2024-09-17T00:32:43.895924+0000 front 2024-09-17T00:32:43.895819+0000 (oldest deadline 2024-09-17T00:33:07.395702+0000) 2024-09-17T00:33:13.717 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:33:13 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-7[66902]: 2024-09-17T00:33:13.322+0000 7f5a4e5b9640 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-09-17T00:32:33.676078+0000 front 2024-09-17T00:32:33.676062+0000 (oldest deadline 2024-09-17T00:32:57.775877+0000) 2024-09-17T00:33:13.717 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:33:13 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-7[66902]: 2024-09-17T00:33:13.322+0000 7f5a4e5b9640 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.2:6814 osd.1 since back 2024-09-17T00:32:44.777292+0000 front 2024-09-17T00:32:44.777224+0000 (oldest deadline 2024-09-17T00:33:08.277122+0000) 2024-09-17T00:33:14.217 INFO:journalctl@ceph.osd.6.smithi099.stdout:Sep 17 00:33:13 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-6[64592]: 2024-09-17T00:33:13.799+0000 7f839b3b6640 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-09-17T00:32:34.544008+0000 front 2024-09-17T00:32:34.544050+0000 (oldest deadline 2024-09-17T00:32:59.243805+0000) 2024-09-17T00:33:14.217 INFO:journalctl@ceph.osd.6.smithi099.stdout:Sep 17 00:33:13 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-6[64592]: 2024-09-17T00:33:13.799+0000 7f839b3b6640 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.2:6814 osd.1 since back 2024-09-17T00:32:39.244223+0000 front 2024-09-17T00:32:39.244308+0000 (oldest deadline 2024-09-17T00:33:09.844982+0000) 2024-09-17T00:33:14.217 INFO:journalctl@ceph.osd.5.smithi099.stdout:Sep 17 00:33:13 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-5[62260]: 2024-09-17T00:33:13.794+0000 7f186ef88640 -1 osd.5 153 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-09-17T00:32:36.894841+0000 front 2024-09-17T00:32:36.894795+0000 (oldest deadline 2024-09-17T00:32:57.994681+0000) 2024-09-17T00:33:14.217 INFO:journalctl@ceph.osd.5.smithi099.stdout:Sep 17 00:33:13 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-5[62260]: 2024-09-17T00:33:13.794+0000 7f186ef88640 -1 osd.5 153 heartbeat_check: no reply from 172.21.15.2:6814 osd.1 since back 2024-09-17T00:32:43.895924+0000 front 2024-09-17T00:32:43.895819+0000 (oldest deadline 2024-09-17T00:33:07.395702+0000) 2024-09-17T00:33:14.640 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Sep 17 00:33:14 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-alertmanager-a[58000]: ts=2024-09-17T00:33:14.239Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi099.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.99:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi002.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.2:8443: connect: connection refused" 2024-09-17T00:33:14.640 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Sep 17 00:33:14 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-alertmanager-a[58000]: ts=2024-09-17T00:33:14.240Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi002.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.2:8443: connect: connection refused" 2024-09-17T00:33:14.640 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Sep 17 00:33:14 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-alertmanager-a[58000]: ts=2024-09-17T00:33:14.241Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi099.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.99:8443: connect: connection refused" 2024-09-17T00:33:14.717 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:33:14 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-7[66902]: 2024-09-17T00:33:14.277+0000 7f5a4e5b9640 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-09-17T00:32:33.676078+0000 front 2024-09-17T00:32:33.676062+0000 (oldest deadline 2024-09-17T00:32:57.775877+0000) 2024-09-17T00:33:14.717 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:33:14 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-7[66902]: 2024-09-17T00:33:14.277+0000 7f5a4e5b9640 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.2:6814 osd.1 since back 2024-09-17T00:32:44.777292+0000 front 2024-09-17T00:32:44.777224+0000 (oldest deadline 2024-09-17T00:33:08.277122+0000) 2024-09-17T00:33:15.217 INFO:journalctl@ceph.osd.6.smithi099.stdout:Sep 17 00:33:14 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-6[64592]: 2024-09-17T00:33:14.825+0000 7f839b3b6640 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-09-17T00:32:34.544008+0000 front 2024-09-17T00:32:34.544050+0000 (oldest deadline 2024-09-17T00:32:59.243805+0000) 2024-09-17T00:33:15.217 INFO:journalctl@ceph.osd.6.smithi099.stdout:Sep 17 00:33:14 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-6[64592]: 2024-09-17T00:33:14.825+0000 7f839b3b6640 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.2:6814 osd.1 since back 2024-09-17T00:32:39.244223+0000 front 2024-09-17T00:32:39.244308+0000 (oldest deadline 2024-09-17T00:33:09.844982+0000) 2024-09-17T00:33:15.217 INFO:journalctl@ceph.osd.5.smithi099.stdout:Sep 17 00:33:14 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-5[62260]: 2024-09-17T00:33:14.807+0000 7f186ef88640 -1 osd.5 153 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-09-17T00:32:36.894841+0000 front 2024-09-17T00:32:36.894795+0000 (oldest deadline 2024-09-17T00:32:57.994681+0000) 2024-09-17T00:33:15.217 INFO:journalctl@ceph.osd.5.smithi099.stdout:Sep 17 00:33:14 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-5[62260]: 2024-09-17T00:33:14.807+0000 7f186ef88640 -1 osd.5 153 heartbeat_check: no reply from 172.21.15.2:6814 osd.1 since back 2024-09-17T00:32:43.895924+0000 front 2024-09-17T00:32:43.895819+0000 (oldest deadline 2024-09-17T00:33:07.395702+0000) 2024-09-17T00:33:15.717 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:33:15 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-7[66902]: 2024-09-17T00:33:15.284+0000 7f5a4e5b9640 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-09-17T00:32:33.676078+0000 front 2024-09-17T00:32:33.676062+0000 (oldest deadline 2024-09-17T00:32:57.775877+0000) 2024-09-17T00:33:15.717 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:33:15 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-7[66902]: 2024-09-17T00:33:15.284+0000 7f5a4e5b9640 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.2:6814 osd.1 since back 2024-09-17T00:32:44.777292+0000 front 2024-09-17T00:32:44.777224+0000 (oldest deadline 2024-09-17T00:33:08.277122+0000) 2024-09-17T00:33:16.217 INFO:journalctl@ceph.osd.5.smithi099.stdout:Sep 17 00:33:15 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-5[62260]: 2024-09-17T00:33:15.834+0000 7f186ef88640 -1 osd.5 153 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-09-17T00:32:36.894841+0000 front 2024-09-17T00:32:36.894795+0000 (oldest deadline 2024-09-17T00:32:57.994681+0000) 2024-09-17T00:33:16.217 INFO:journalctl@ceph.osd.5.smithi099.stdout:Sep 17 00:33:15 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-5[62260]: 2024-09-17T00:33:15.834+0000 7f186ef88640 -1 osd.5 153 heartbeat_check: no reply from 172.21.15.2:6814 osd.1 since back 2024-09-17T00:32:43.895924+0000 front 2024-09-17T00:32:43.895819+0000 (oldest deadline 2024-09-17T00:33:07.395702+0000) 2024-09-17T00:33:16.217 INFO:journalctl@ceph.osd.6.smithi099.stdout:Sep 17 00:33:15 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-6[64592]: 2024-09-17T00:33:15.835+0000 7f839b3b6640 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-09-17T00:32:34.544008+0000 front 2024-09-17T00:32:34.544050+0000 (oldest deadline 2024-09-17T00:32:59.243805+0000) 2024-09-17T00:33:16.217 INFO:journalctl@ceph.osd.6.smithi099.stdout:Sep 17 00:33:15 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-6[64592]: 2024-09-17T00:33:15.835+0000 7f839b3b6640 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.2:6814 osd.1 since back 2024-09-17T00:32:39.244223+0000 front 2024-09-17T00:32:39.244308+0000 (oldest deadline 2024-09-17T00:33:09.844982+0000) 2024-09-17T00:33:16.217 INFO:journalctl@ceph.osd.6.smithi099.stdout:Sep 17 00:33:15 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-6[64592]: 2024-09-17T00:33:15.835+0000 7f839b3b6640 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.2:6822 osd.2 since back 2024-09-17T00:32:49.845377+0000 front 2024-09-17T00:32:49.845330+0000 (oldest deadline 2024-09-17T00:33:15.745300+0000) 2024-09-17T00:33:16.717 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:33:16 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-7[66902]: 2024-09-17T00:33:16.269+0000 7f5a4e5b9640 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-09-17T00:32:33.676078+0000 front 2024-09-17T00:32:33.676062+0000 (oldest deadline 2024-09-17T00:32:57.775877+0000) 2024-09-17T00:33:16.717 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:33:16 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-7[66902]: 2024-09-17T00:33:16.269+0000 7f5a4e5b9640 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.2:6814 osd.1 since back 2024-09-17T00:32:44.777292+0000 front 2024-09-17T00:32:44.777224+0000 (oldest deadline 2024-09-17T00:33:08.277122+0000) 2024-09-17T00:33:17.217 INFO:journalctl@ceph.osd.5.smithi099.stdout:Sep 17 00:33:16 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-5[62260]: 2024-09-17T00:33:16.800+0000 7f186ef88640 -1 osd.5 153 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-09-17T00:32:36.894841+0000 front 2024-09-17T00:32:36.894795+0000 (oldest deadline 2024-09-17T00:32:57.994681+0000) 2024-09-17T00:33:17.217 INFO:journalctl@ceph.osd.5.smithi099.stdout:Sep 17 00:33:16 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-5[62260]: 2024-09-17T00:33:16.800+0000 7f186ef88640 -1 osd.5 153 heartbeat_check: no reply from 172.21.15.2:6814 osd.1 since back 2024-09-17T00:32:43.895924+0000 front 2024-09-17T00:32:43.895819+0000 (oldest deadline 2024-09-17T00:33:07.395702+0000) 2024-09-17T00:33:17.217 INFO:journalctl@ceph.osd.5.smithi099.stdout:Sep 17 00:33:16 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-5[62260]: 2024-09-17T00:33:16.800+0000 7f186ef88640 -1 osd.5 153 heartbeat_check: no reply from 172.21.15.2:6822 osd.2 since back 2024-09-17T00:32:47.396170+0000 front 2024-09-17T00:32:47.396188+0000 (oldest deadline 2024-09-17T00:33:16.196558+0000) 2024-09-17T00:33:17.218 INFO:journalctl@ceph.osd.6.smithi099.stdout:Sep 17 00:33:16 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-6[64592]: 2024-09-17T00:33:16.811+0000 7f839b3b6640 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-09-17T00:32:34.544008+0000 front 2024-09-17T00:32:34.544050+0000 (oldest deadline 2024-09-17T00:32:59.243805+0000) 2024-09-17T00:33:17.218 INFO:journalctl@ceph.osd.6.smithi099.stdout:Sep 17 00:33:16 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-6[64592]: 2024-09-17T00:33:16.811+0000 7f839b3b6640 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.2:6814 osd.1 since back 2024-09-17T00:32:39.244223+0000 front 2024-09-17T00:32:39.244308+0000 (oldest deadline 2024-09-17T00:33:09.844982+0000) 2024-09-17T00:33:17.218 INFO:journalctl@ceph.osd.6.smithi099.stdout:Sep 17 00:33:16 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-6[64592]: 2024-09-17T00:33:16.811+0000 7f839b3b6640 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.2:6822 osd.2 since back 2024-09-17T00:32:49.845377+0000 front 2024-09-17T00:32:49.845330+0000 (oldest deadline 2024-09-17T00:33:15.745300+0000) 2024-09-17T00:33:17.565 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:33:17 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-7[66902]: 2024-09-17T00:33:17.253+0000 7f5a4e5b9640 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-09-17T00:32:33.676078+0000 front 2024-09-17T00:32:33.676062+0000 (oldest deadline 2024-09-17T00:32:57.775877+0000) 2024-09-17T00:33:17.565 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:33:17 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-7[66902]: 2024-09-17T00:33:17.253+0000 7f5a4e5b9640 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.2:6814 osd.1 since back 2024-09-17T00:32:44.777292+0000 front 2024-09-17T00:32:44.777224+0000 (oldest deadline 2024-09-17T00:33:08.277122+0000) 2024-09-17T00:33:17.825 INFO:journalctl@ceph.osd.5.smithi099.stdout:Sep 17 00:33:17 smithi099 podman[75588]: 2024-09-17 00:33:17.568776257 +0000 UTC m=+5.619363751 container died 2df0d88d16ce71ff763e894d8734cd09ebe3c12c64772f073d54151a0df05394 (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-c2ddf97, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-17T00:33:18.217 INFO:journalctl@ceph.osd.5.smithi099.stdout:Sep 17 00:33:18 smithi099 podman[75588]: 2024-09-17 00:33:18.013872428 +0000 UTC m=+6.064459964 container cleanup 2df0d88d16ce71ff763e894d8734cd09ebe3c12c64772f073d54151a0df05394 (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-5, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, RELEASE=squid-c2ddf97) 2024-09-17T00:33:18.217 INFO:journalctl@ceph.osd.5.smithi099.stdout:Sep 17 00:33:18 smithi099 bash[75588]: ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-5 2024-09-17T00:33:18.217 INFO:journalctl@ceph.osd.6.smithi099.stdout:Sep 17 00:33:17 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-6[64592]: 2024-09-17T00:33:17.828+0000 7f839b3b6640 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-09-17T00:32:34.544008+0000 front 2024-09-17T00:32:34.544050+0000 (oldest deadline 2024-09-17T00:32:59.243805+0000) 2024-09-17T00:33:18.218 INFO:journalctl@ceph.osd.6.smithi099.stdout:Sep 17 00:33:17 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-6[64592]: 2024-09-17T00:33:17.828+0000 7f839b3b6640 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.2:6814 osd.1 since back 2024-09-17T00:32:39.244223+0000 front 2024-09-17T00:32:39.244308+0000 (oldest deadline 2024-09-17T00:33:09.844982+0000) 2024-09-17T00:33:18.218 INFO:journalctl@ceph.osd.6.smithi099.stdout:Sep 17 00:33:17 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-6[64592]: 2024-09-17T00:33:17.828+0000 7f839b3b6640 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.2:6822 osd.2 since back 2024-09-17T00:32:49.845377+0000 front 2024-09-17T00:32:49.845330+0000 (oldest deadline 2024-09-17T00:33:15.745300+0000) 2024-09-17T00:33:18.717 INFO:journalctl@ceph.osd.5.smithi099.stdout:Sep 17 00:33:18 smithi099 podman[75605]: 2024-09-17 00:33:18.407648585 +0000 UTC m=+0.837432540 container remove 2df0d88d16ce71ff763e894d8734cd09ebe3c12c64772f073d54151a0df05394 (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-5, RELEASE=squid-c2ddf97, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2) 2024-09-17T00:33:18.717 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:33:18 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-7[66902]: 2024-09-17T00:33:18.237+0000 7f5a4e5b9640 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-09-17T00:32:33.676078+0000 front 2024-09-17T00:32:33.676062+0000 (oldest deadline 2024-09-17T00:32:57.775877+0000) 2024-09-17T00:33:18.717 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:33:18 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-7[66902]: 2024-09-17T00:33:18.237+0000 7f5a4e5b9640 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.2:6814 osd.1 since back 2024-09-17T00:32:44.777292+0000 front 2024-09-17T00:32:44.777224+0000 (oldest deadline 2024-09-17T00:33:08.277122+0000) 2024-09-17T00:33:18.718 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:33:18 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-7[66902]: 2024-09-17T00:33:18.237+0000 7f5a4e5b9640 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.2:6822 osd.2 since back 2024-09-17T00:32:52.377964+0000 front 2024-09-17T00:32:52.378040+0000 (oldest deadline 2024-09-17T00:33:17.677876+0000) 2024-09-17T00:33:19.041 INFO:journalctl@ceph.osd.5.smithi099.stdout:Sep 17 00:33:18 smithi099 podman[75696]: 2024-09-17 00:33:18.685353162 +0000 UTC m=+0.015917323 image pull ebb0615149def4a3179c82c0cc47af75808300abb9f1142788e5e605a7ac0e5b quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f 2024-09-17T00:33:19.041 INFO:journalctl@ceph.osd.5.smithi099.stdout:Sep 17 00:33:18 smithi099 podman[75696]: 2024-09-17 00:33:18.817067535 +0000 UTC m=+0.147631682 container create 0980324ac1df6e807e4500a37f0f687c42ce490b5f1206f94114fff9006eb015 (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-5-deactivate, RELEASE=squid-c2ddf97, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-17T00:33:19.042 INFO:journalctl@ceph.osd.6.smithi099.stdout:Sep 17 00:33:18 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-6[64592]: 2024-09-17T00:33:18.851+0000 7f839b3b6640 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-09-17T00:32:34.544008+0000 front 2024-09-17T00:32:34.544050+0000 (oldest deadline 2024-09-17T00:32:59.243805+0000) 2024-09-17T00:33:19.042 INFO:journalctl@ceph.osd.6.smithi099.stdout:Sep 17 00:33:18 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-6[64592]: 2024-09-17T00:33:18.851+0000 7f839b3b6640 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.2:6814 osd.1 since back 2024-09-17T00:32:39.244223+0000 front 2024-09-17T00:32:39.244308+0000 (oldest deadline 2024-09-17T00:33:09.844982+0000) 2024-09-17T00:33:19.042 INFO:journalctl@ceph.osd.6.smithi099.stdout:Sep 17 00:33:18 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-6[64592]: 2024-09-17T00:33:18.851+0000 7f839b3b6640 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.2:6822 osd.2 since back 2024-09-17T00:32:49.845377+0000 front 2024-09-17T00:32:49.845330+0000 (oldest deadline 2024-09-17T00:33:15.745300+0000) 2024-09-17T00:33:19.467 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:33:19 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-7[66902]: 2024-09-17T00:33:19.190+0000 7f5a4e5b9640 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-09-17T00:32:33.676078+0000 front 2024-09-17T00:32:33.676062+0000 (oldest deadline 2024-09-17T00:32:57.775877+0000) 2024-09-17T00:33:19.467 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:33:19 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-7[66902]: 2024-09-17T00:33:19.190+0000 7f5a4e5b9640 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.2:6814 osd.1 since back 2024-09-17T00:32:44.777292+0000 front 2024-09-17T00:32:44.777224+0000 (oldest deadline 2024-09-17T00:33:08.277122+0000) 2024-09-17T00:33:19.467 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:33:19 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-7[66902]: 2024-09-17T00:33:19.190+0000 7f5a4e5b9640 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.2:6822 osd.2 since back 2024-09-17T00:32:52.377964+0000 front 2024-09-17T00:32:52.378040+0000 (oldest deadline 2024-09-17T00:33:17.677876+0000) 2024-09-17T00:33:19.467 INFO:journalctl@ceph.osd.5.smithi099.stdout:Sep 17 00:33:19 smithi099 podman[75696]: 2024-09-17 00:33:19.044641139 +0000 UTC m=+0.375205297 container init 0980324ac1df6e807e4500a37f0f687c42ce490b5f1206f94114fff9006eb015 (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-5-deactivate, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, RELEASE=squid-c2ddf97, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-09-17T00:33:19.467 INFO:journalctl@ceph.osd.5.smithi099.stdout:Sep 17 00:33:19 smithi099 podman[75696]: 2024-09-17 00:33:19.047910324 +0000 UTC m=+0.378474473 container start 0980324ac1df6e807e4500a37f0f687c42ce490b5f1206f94114fff9006eb015 (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-5-deactivate, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-c2ddf97, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-09-17T00:33:19.468 INFO:journalctl@ceph.osd.5.smithi099.stdout:Sep 17 00:33:19 smithi099 podman[75696]: 2024-09-17 00:33:19.13649719 +0000 UTC m=+0.467061336 container attach 0980324ac1df6e807e4500a37f0f687c42ce490b5f1206f94114fff9006eb015 (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-5-deactivate, org.label-schema.schema-version=1.0, RELEASE=squid-c2ddf97, io.buildah.version=1.37.2, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-17T00:33:19.468 INFO:journalctl@ceph.osd.5.smithi099.stdout:Sep 17 00:33:19 smithi099 podman[75696]: 2024-09-17 00:33:19.23887948 +0000 UTC m=+0.569443640 container died 0980324ac1df6e807e4500a37f0f687c42ce490b5f1206f94114fff9006eb015 (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-5-deactivate, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, RELEASE=squid-c2ddf97, org.label-schema.vendor=CentOS) 2024-09-17T00:33:19.860 DEBUG:teuthology.orchestra.run.smithi099:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3aad27a2-7489-11ef-bceb-c7b262605968@osd.5.service' 2024-09-17T00:33:19.901 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-17T00:33:19.901 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-09-17T00:33:19.901 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-09-17T00:33:19.901 DEBUG:teuthology.orchestra.run.smithi099:> sudo systemctl stop ceph-3aad27a2-7489-11ef-bceb-c7b262605968@osd.6 2024-09-17T00:33:20.203 INFO:journalctl@ceph.osd.6.smithi099.stdout:Sep 17 00:33:19 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-6[64592]: 2024-09-17T00:33:19.854+0000 7f839b3b6640 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-09-17T00:32:34.544008+0000 front 2024-09-17T00:32:34.544050+0000 (oldest deadline 2024-09-17T00:32:59.243805+0000) 2024-09-17T00:33:20.204 INFO:journalctl@ceph.osd.6.smithi099.stdout:Sep 17 00:33:19 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-6[64592]: 2024-09-17T00:33:19.854+0000 7f839b3b6640 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.2:6814 osd.1 since back 2024-09-17T00:32:39.244223+0000 front 2024-09-17T00:32:39.244308+0000 (oldest deadline 2024-09-17T00:33:09.844982+0000) 2024-09-17T00:33:20.204 INFO:journalctl@ceph.osd.6.smithi099.stdout:Sep 17 00:33:19 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-6[64592]: 2024-09-17T00:33:19.854+0000 7f839b3b6640 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.2:6822 osd.2 since back 2024-09-17T00:32:49.845377+0000 front 2024-09-17T00:32:49.845330+0000 (oldest deadline 2024-09-17T00:33:15.745300+0000) 2024-09-17T00:33:20.204 INFO:journalctl@ceph.osd.6.smithi099.stdout:Sep 17 00:33:19 smithi099 systemd[1]: Stopping Ceph osd.6 for 3aad27a2-7489-11ef-bceb-c7b262605968... 2024-09-17T00:33:20.467 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:33:20 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-7[66902]: 2024-09-17T00:33:20.206+0000 7f5a4e5b9640 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-09-17T00:32:33.676078+0000 front 2024-09-17T00:32:33.676062+0000 (oldest deadline 2024-09-17T00:32:57.775877+0000) 2024-09-17T00:33:20.467 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:33:20 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-7[66902]: 2024-09-17T00:33:20.206+0000 7f5a4e5b9640 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.2:6814 osd.1 since back 2024-09-17T00:32:44.777292+0000 front 2024-09-17T00:32:44.777224+0000 (oldest deadline 2024-09-17T00:33:08.277122+0000) 2024-09-17T00:33:20.467 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:33:20 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-7[66902]: 2024-09-17T00:33:20.206+0000 7f5a4e5b9640 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.2:6822 osd.2 since back 2024-09-17T00:32:52.377964+0000 front 2024-09-17T00:32:52.378040+0000 (oldest deadline 2024-09-17T00:33:17.677876+0000) 2024-09-17T00:33:20.467 INFO:journalctl@ceph.osd.6.smithi099.stdout:Sep 17 00:33:20 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-6[64592]: 2024-09-17T00:33:20.251+0000 7f839eda0640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-17T00:33:20.467 INFO:journalctl@ceph.osd.6.smithi099.stdout:Sep 17 00:33:20 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-6[64592]: 2024-09-17T00:33:20.251+0000 7f839eda0640 -1 osd.6 153 *** Got signal Terminated *** 2024-09-17T00:33:20.467 INFO:journalctl@ceph.osd.6.smithi099.stdout:Sep 17 00:33:20 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-6[64592]: 2024-09-17T00:33:20.251+0000 7f839eda0640 -1 osd.6 153 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-17T00:33:21.201 INFO:journalctl@ceph.osd.6.smithi099.stdout:Sep 17 00:33:20 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-6[64592]: 2024-09-17T00:33:20.832+0000 7f839b3b6640 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-09-17T00:32:34.544008+0000 front 2024-09-17T00:32:34.544050+0000 (oldest deadline 2024-09-17T00:32:59.243805+0000) 2024-09-17T00:33:21.201 INFO:journalctl@ceph.osd.6.smithi099.stdout:Sep 17 00:33:20 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-6[64592]: 2024-09-17T00:33:20.832+0000 7f839b3b6640 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.2:6814 osd.1 since back 2024-09-17T00:32:39.244223+0000 front 2024-09-17T00:32:39.244308+0000 (oldest deadline 2024-09-17T00:33:09.844982+0000) 2024-09-17T00:33:21.201 INFO:journalctl@ceph.osd.6.smithi099.stdout:Sep 17 00:33:20 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-6[64592]: 2024-09-17T00:33:20.832+0000 7f839b3b6640 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.2:6822 osd.2 since back 2024-09-17T00:32:49.845377+0000 front 2024-09-17T00:32:49.845330+0000 (oldest deadline 2024-09-17T00:33:15.745300+0000) 2024-09-17T00:33:21.467 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:33:21 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-7[66902]: 2024-09-17T00:33:21.203+0000 7f5a4e5b9640 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-09-17T00:32:33.676078+0000 front 2024-09-17T00:32:33.676062+0000 (oldest deadline 2024-09-17T00:32:57.775877+0000) 2024-09-17T00:33:21.467 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:33:21 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-7[66902]: 2024-09-17T00:33:21.203+0000 7f5a4e5b9640 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.2:6814 osd.1 since back 2024-09-17T00:32:44.777292+0000 front 2024-09-17T00:32:44.777224+0000 (oldest deadline 2024-09-17T00:33:08.277122+0000) 2024-09-17T00:33:21.467 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:33:21 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-7[66902]: 2024-09-17T00:33:21.203+0000 7f5a4e5b9640 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.2:6822 osd.2 since back 2024-09-17T00:32:52.377964+0000 front 2024-09-17T00:32:52.378040+0000 (oldest deadline 2024-09-17T00:33:17.677876+0000) 2024-09-17T00:33:22.173 INFO:journalctl@ceph.osd.6.smithi099.stdout:Sep 17 00:33:21 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-6[64592]: 2024-09-17T00:33:21.803+0000 7f839b3b6640 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-09-17T00:32:34.544008+0000 front 2024-09-17T00:32:34.544050+0000 (oldest deadline 2024-09-17T00:32:59.243805+0000) 2024-09-17T00:33:22.173 INFO:journalctl@ceph.osd.6.smithi099.stdout:Sep 17 00:33:21 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-6[64592]: 2024-09-17T00:33:21.803+0000 7f839b3b6640 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.2:6814 osd.1 since back 2024-09-17T00:32:39.244223+0000 front 2024-09-17T00:32:39.244308+0000 (oldest deadline 2024-09-17T00:33:09.844982+0000) 2024-09-17T00:33:22.173 INFO:journalctl@ceph.osd.6.smithi099.stdout:Sep 17 00:33:21 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-6[64592]: 2024-09-17T00:33:21.803+0000 7f839b3b6640 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.2:6822 osd.2 since back 2024-09-17T00:32:49.845377+0000 front 2024-09-17T00:32:49.845330+0000 (oldest deadline 2024-09-17T00:33:15.745300+0000) 2024-09-17T00:33:22.467 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:33:22 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-7[66902]: 2024-09-17T00:33:22.174+0000 7f5a4e5b9640 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-09-17T00:32:33.676078+0000 front 2024-09-17T00:32:33.676062+0000 (oldest deadline 2024-09-17T00:32:57.775877+0000) 2024-09-17T00:33:22.467 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:33:22 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-7[66902]: 2024-09-17T00:33:22.174+0000 7f5a4e5b9640 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.2:6814 osd.1 since back 2024-09-17T00:32:44.777292+0000 front 2024-09-17T00:32:44.777224+0000 (oldest deadline 2024-09-17T00:33:08.277122+0000) 2024-09-17T00:33:22.467 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:33:22 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-7[66902]: 2024-09-17T00:33:22.174+0000 7f5a4e5b9640 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.2:6822 osd.2 since back 2024-09-17T00:32:52.377964+0000 front 2024-09-17T00:32:52.378040+0000 (oldest deadline 2024-09-17T00:33:17.677876+0000) 2024-09-17T00:33:22.467 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:33:22 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-7[66902]: 2024-09-17T00:33:22.174+0000 7f5a4e5b9640 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.2:6830 osd.3 since back 2024-09-17T00:33:00.478968+0000 front 2024-09-17T00:33:00.478933+0000 (oldest deadline 2024-09-17T00:33:21.578873+0000) 2024-09-17T00:33:23.130 INFO:journalctl@ceph.osd.6.smithi099.stdout:Sep 17 00:33:22 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-6[64592]: 2024-09-17T00:33:22.831+0000 7f839b3b6640 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-09-17T00:32:34.544008+0000 front 2024-09-17T00:32:34.544050+0000 (oldest deadline 2024-09-17T00:32:59.243805+0000) 2024-09-17T00:33:23.130 INFO:journalctl@ceph.osd.6.smithi099.stdout:Sep 17 00:33:22 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-6[64592]: 2024-09-17T00:33:22.831+0000 7f839b3b6640 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.2:6814 osd.1 since back 2024-09-17T00:32:39.244223+0000 front 2024-09-17T00:32:39.244308+0000 (oldest deadline 2024-09-17T00:33:09.844982+0000) 2024-09-17T00:33:23.130 INFO:journalctl@ceph.osd.6.smithi099.stdout:Sep 17 00:33:22 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-6[64592]: 2024-09-17T00:33:22.831+0000 7f839b3b6640 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.2:6822 osd.2 since back 2024-09-17T00:32:49.845377+0000 front 2024-09-17T00:32:49.845330+0000 (oldest deadline 2024-09-17T00:33:15.745300+0000) 2024-09-17T00:33:23.467 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:33:23 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-7[66902]: 2024-09-17T00:33:23.132+0000 7f5a4e5b9640 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-09-17T00:32:33.676078+0000 front 2024-09-17T00:32:33.676062+0000 (oldest deadline 2024-09-17T00:32:57.775877+0000) 2024-09-17T00:33:23.467 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:33:23 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-7[66902]: 2024-09-17T00:33:23.132+0000 7f5a4e5b9640 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.2:6814 osd.1 since back 2024-09-17T00:32:44.777292+0000 front 2024-09-17T00:32:44.777224+0000 (oldest deadline 2024-09-17T00:33:08.277122+0000) 2024-09-17T00:33:23.467 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:33:23 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-7[66902]: 2024-09-17T00:33:23.132+0000 7f5a4e5b9640 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.2:6822 osd.2 since back 2024-09-17T00:32:52.377964+0000 front 2024-09-17T00:32:52.378040+0000 (oldest deadline 2024-09-17T00:33:17.677876+0000) 2024-09-17T00:33:23.467 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:33:23 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-7[66902]: 2024-09-17T00:33:23.132+0000 7f5a4e5b9640 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.2:6830 osd.3 since back 2024-09-17T00:33:00.478968+0000 front 2024-09-17T00:33:00.478933+0000 (oldest deadline 2024-09-17T00:33:21.578873+0000) 2024-09-17T00:33:24.125 INFO:journalctl@ceph.osd.6.smithi099.stdout:Sep 17 00:33:23 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-6[64592]: 2024-09-17T00:33:23.797+0000 7f839b3b6640 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-09-17T00:32:34.544008+0000 front 2024-09-17T00:32:34.544050+0000 (oldest deadline 2024-09-17T00:32:59.243805+0000) 2024-09-17T00:33:24.128 INFO:journalctl@ceph.osd.6.smithi099.stdout:Sep 17 00:33:23 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-6[64592]: 2024-09-17T00:33:23.797+0000 7f839b3b6640 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.2:6814 osd.1 since back 2024-09-17T00:32:39.244223+0000 front 2024-09-17T00:32:39.244308+0000 (oldest deadline 2024-09-17T00:33:09.844982+0000) 2024-09-17T00:33:24.128 INFO:journalctl@ceph.osd.6.smithi099.stdout:Sep 17 00:33:23 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-6[64592]: 2024-09-17T00:33:23.797+0000 7f839b3b6640 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.2:6822 osd.2 since back 2024-09-17T00:32:49.845377+0000 front 2024-09-17T00:32:49.845330+0000 (oldest deadline 2024-09-17T00:33:15.745300+0000) 2024-09-17T00:33:24.128 INFO:journalctl@ceph.osd.6.smithi099.stdout:Sep 17 00:33:23 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-6[64592]: 2024-09-17T00:33:23.797+0000 7f839b3b6640 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.2:6830 osd.3 since back 2024-09-17T00:32:59.246162+0000 front 2024-09-17T00:32:59.246139+0000 (oldest deadline 2024-09-17T00:33:23.346072+0000) 2024-09-17T00:33:24.467 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:33:24 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-7[66902]: 2024-09-17T00:33:24.126+0000 7f5a4e5b9640 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-09-17T00:32:33.676078+0000 front 2024-09-17T00:32:33.676062+0000 (oldest deadline 2024-09-17T00:32:57.775877+0000) 2024-09-17T00:33:24.467 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:33:24 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-7[66902]: 2024-09-17T00:33:24.126+0000 7f5a4e5b9640 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.2:6814 osd.1 since back 2024-09-17T00:32:44.777292+0000 front 2024-09-17T00:32:44.777224+0000 (oldest deadline 2024-09-17T00:33:08.277122+0000) 2024-09-17T00:33:24.467 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:33:24 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-7[66902]: 2024-09-17T00:33:24.126+0000 7f5a4e5b9640 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.2:6822 osd.2 since back 2024-09-17T00:32:52.377964+0000 front 2024-09-17T00:32:52.378040+0000 (oldest deadline 2024-09-17T00:33:17.677876+0000) 2024-09-17T00:33:24.467 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:33:24 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-7[66902]: 2024-09-17T00:33:24.126+0000 7f5a4e5b9640 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.2:6830 osd.3 since back 2024-09-17T00:33:00.478968+0000 front 2024-09-17T00:33:00.478933+0000 (oldest deadline 2024-09-17T00:33:21.578873+0000) 2024-09-17T00:33:24.729 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Sep 17 00:33:24 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-alertmanager-a[58000]: ts=2024-09-17T00:33:24.241Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi002.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.2:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi099.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.99:8443: connect: connection refused" 2024-09-17T00:33:24.730 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Sep 17 00:33:24 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-alertmanager-a[58000]: ts=2024-09-17T00:33:24.241Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi002.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.2:8443: connect: connection refused" 2024-09-17T00:33:24.730 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Sep 17 00:33:24 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-alertmanager-a[58000]: ts=2024-09-17T00:33:24.242Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi099.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.99:8443: connect: connection refused" 2024-09-17T00:33:25.107 INFO:journalctl@ceph.osd.6.smithi099.stdout:Sep 17 00:33:24 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-6[64592]: 2024-09-17T00:33:24.783+0000 7f839b3b6640 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-09-17T00:32:34.544008+0000 front 2024-09-17T00:32:34.544050+0000 (oldest deadline 2024-09-17T00:32:59.243805+0000) 2024-09-17T00:33:25.108 INFO:journalctl@ceph.osd.6.smithi099.stdout:Sep 17 00:33:24 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-6[64592]: 2024-09-17T00:33:24.783+0000 7f839b3b6640 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.2:6814 osd.1 since back 2024-09-17T00:32:39.244223+0000 front 2024-09-17T00:32:39.244308+0000 (oldest deadline 2024-09-17T00:33:09.844982+0000) 2024-09-17T00:33:25.108 INFO:journalctl@ceph.osd.6.smithi099.stdout:Sep 17 00:33:24 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-6[64592]: 2024-09-17T00:33:24.783+0000 7f839b3b6640 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.2:6822 osd.2 since back 2024-09-17T00:32:49.845377+0000 front 2024-09-17T00:32:49.845330+0000 (oldest deadline 2024-09-17T00:33:15.745300+0000) 2024-09-17T00:33:25.108 INFO:journalctl@ceph.osd.6.smithi099.stdout:Sep 17 00:33:24 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-6[64592]: 2024-09-17T00:33:24.783+0000 7f839b3b6640 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.2:6830 osd.3 since back 2024-09-17T00:32:59.246162+0000 front 2024-09-17T00:32:59.246139+0000 (oldest deadline 2024-09-17T00:33:23.346072+0000) 2024-09-17T00:33:25.467 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:33:25 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-7[66902]: 2024-09-17T00:33:25.109+0000 7f5a4e5b9640 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-09-17T00:32:33.676078+0000 front 2024-09-17T00:32:33.676062+0000 (oldest deadline 2024-09-17T00:32:57.775877+0000) 2024-09-17T00:33:25.467 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:33:25 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-7[66902]: 2024-09-17T00:33:25.109+0000 7f5a4e5b9640 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.2:6814 osd.1 since back 2024-09-17T00:32:44.777292+0000 front 2024-09-17T00:32:44.777224+0000 (oldest deadline 2024-09-17T00:33:08.277122+0000) 2024-09-17T00:33:25.467 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:33:25 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-7[66902]: 2024-09-17T00:33:25.109+0000 7f5a4e5b9640 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.2:6822 osd.2 since back 2024-09-17T00:32:52.377964+0000 front 2024-09-17T00:32:52.378040+0000 (oldest deadline 2024-09-17T00:33:17.677876+0000) 2024-09-17T00:33:25.467 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:33:25 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-7[66902]: 2024-09-17T00:33:25.109+0000 7f5a4e5b9640 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.2:6830 osd.3 since back 2024-09-17T00:33:00.478968+0000 front 2024-09-17T00:33:00.478933+0000 (oldest deadline 2024-09-17T00:33:21.578873+0000) 2024-09-17T00:33:25.924 INFO:journalctl@ceph.osd.6.smithi099.stdout:Sep 17 00:33:25 smithi099 podman[75799]: 2024-09-17 00:33:25.662952108 +0000 UTC m=+5.527991789 container died 7462bdbdf29b7ec691c9bfa3dc9777f167fc9ab0c3a8e6663e289c2a3ca8f1bd (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-6, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-c2ddf97, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux ) 2024-09-17T00:33:26.217 INFO:journalctl@ceph.osd.6.smithi099.stdout:Sep 17 00:33:26 smithi099 podman[75799]: 2024-09-17 00:33:26.109500637 +0000 UTC m=+5.974540316 container cleanup 7462bdbdf29b7ec691c9bfa3dc9777f167fc9ab0c3a8e6663e289c2a3ca8f1bd (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-6, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=squid-c2ddf97, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731) 2024-09-17T00:33:26.217 INFO:journalctl@ceph.osd.6.smithi099.stdout:Sep 17 00:33:26 smithi099 bash[75799]: ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-6 2024-09-17T00:33:26.217 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:33:26 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-7[66902]: 2024-09-17T00:33:26.152+0000 7f5a4e5b9640 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-09-17T00:32:33.676078+0000 front 2024-09-17T00:32:33.676062+0000 (oldest deadline 2024-09-17T00:32:57.775877+0000) 2024-09-17T00:33:26.217 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:33:26 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-7[66902]: 2024-09-17T00:33:26.152+0000 7f5a4e5b9640 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.2:6814 osd.1 since back 2024-09-17T00:32:44.777292+0000 front 2024-09-17T00:32:44.777224+0000 (oldest deadline 2024-09-17T00:33:08.277122+0000) 2024-09-17T00:33:26.217 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:33:26 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-7[66902]: 2024-09-17T00:33:26.152+0000 7f5a4e5b9640 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.2:6822 osd.2 since back 2024-09-17T00:32:52.377964+0000 front 2024-09-17T00:32:52.378040+0000 (oldest deadline 2024-09-17T00:33:17.677876+0000) 2024-09-17T00:33:26.218 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:33:26 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-7[66902]: 2024-09-17T00:33:26.152+0000 7f5a4e5b9640 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.2:6830 osd.3 since back 2024-09-17T00:33:00.478968+0000 front 2024-09-17T00:33:00.478933+0000 (oldest deadline 2024-09-17T00:33:21.578873+0000) 2024-09-17T00:33:26.717 INFO:journalctl@ceph.osd.6.smithi099.stdout:Sep 17 00:33:26 smithi099 podman[75815]: 2024-09-17 00:33:26.402850556 +0000 UTC m=+0.741211754 container remove 7462bdbdf29b7ec691c9bfa3dc9777f167fc9ab0c3a8e6663e289c2a3ca8f1bd (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-6, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=squid-c2ddf97, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2) 2024-09-17T00:33:27.028 INFO:journalctl@ceph.osd.6.smithi099.stdout:Sep 17 00:33:26 smithi099 podman[75902]: 2024-09-17 00:33:26.624542506 +0000 UTC m=+0.015859141 image pull ebb0615149def4a3179c82c0cc47af75808300abb9f1142788e5e605a7ac0e5b quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f 2024-09-17T00:33:27.028 INFO:journalctl@ceph.osd.6.smithi099.stdout:Sep 17 00:33:26 smithi099 podman[75902]: 2024-09-17 00:33:26.728769537 +0000 UTC m=+0.120086163 container create 03d13e0d6efd59ab8e49aa67ef3f071f6ee5132da7a9cd1c31db0d0222645641 (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-6-deactivate, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-c2ddf97, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True) 2024-09-17T00:33:27.029 INFO:journalctl@ceph.osd.6.smithi099.stdout:Sep 17 00:33:26 smithi099 podman[75902]: 2024-09-17 00:33:26.95287034 +0000 UTC m=+0.344186979 container init 03d13e0d6efd59ab8e49aa67ef3f071f6ee5132da7a9cd1c31db0d0222645641 (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-6-deactivate, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-c2ddf97, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, CEPH_POINT_RELEASE=) 2024-09-17T00:33:27.029 INFO:journalctl@ceph.osd.6.smithi099.stdout:Sep 17 00:33:26 smithi099 podman[75902]: 2024-09-17 00:33:26.95657048 +0000 UTC m=+0.347887116 container start 03d13e0d6efd59ab8e49aa67ef3f071f6ee5132da7a9cd1c31db0d0222645641 (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-6-deactivate, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-c2ddf97, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-17T00:33:27.029 INFO:journalctl@ceph.osd.6.smithi099.stdout:Sep 17 00:33:27 smithi099 podman[75902]: 2024-09-17 00:33:27.031475724 +0000 UTC m=+0.422792371 container attach 03d13e0d6efd59ab8e49aa67ef3f071f6ee5132da7a9cd1c31db0d0222645641 (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-6-deactivate, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=squid-c2ddf97, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-09-17T00:33:27.432 INFO:journalctl@ceph.osd.6.smithi099.stdout:Sep 17 00:33:27 smithi099 podman[75902]: 2024-09-17 00:33:27.140055701 +0000 UTC m=+0.531372332 container died 03d13e0d6efd59ab8e49aa67ef3f071f6ee5132da7a9cd1c31db0d0222645641 (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-6-deactivate, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=squid-c2ddf97, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, ceph=True) 2024-09-17T00:33:27.432 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:33:27 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-7[66902]: 2024-09-17T00:33:27.172+0000 7f5a4e5b9640 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-09-17T00:32:33.676078+0000 front 2024-09-17T00:32:33.676062+0000 (oldest deadline 2024-09-17T00:32:57.775877+0000) 2024-09-17T00:33:27.432 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:33:27 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-7[66902]: 2024-09-17T00:33:27.172+0000 7f5a4e5b9640 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.2:6814 osd.1 since back 2024-09-17T00:32:44.777292+0000 front 2024-09-17T00:32:44.777224+0000 (oldest deadline 2024-09-17T00:33:08.277122+0000) 2024-09-17T00:33:27.433 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:33:27 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-7[66902]: 2024-09-17T00:33:27.172+0000 7f5a4e5b9640 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.2:6822 osd.2 since back 2024-09-17T00:32:52.377964+0000 front 2024-09-17T00:32:52.378040+0000 (oldest deadline 2024-09-17T00:33:17.677876+0000) 2024-09-17T00:33:27.433 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:33:27 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-7[66902]: 2024-09-17T00:33:27.172+0000 7f5a4e5b9640 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.2:6830 osd.3 since back 2024-09-17T00:33:00.478968+0000 front 2024-09-17T00:33:00.478933+0000 (oldest deadline 2024-09-17T00:33:21.578873+0000) 2024-09-17T00:33:27.733 DEBUG:teuthology.orchestra.run.smithi099:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3aad27a2-7489-11ef-bceb-c7b262605968@osd.6.service' 2024-09-17T00:33:27.770 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-17T00:33:27.770 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-09-17T00:33:27.771 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-09-17T00:33:27.771 DEBUG:teuthology.orchestra.run.smithi099:> sudo systemctl stop ceph-3aad27a2-7489-11ef-bceb-c7b262605968@osd.7 2024-09-17T00:33:27.967 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:33:27 smithi099 systemd[1]: Stopping Ceph osd.7 for 3aad27a2-7489-11ef-bceb-c7b262605968... 2024-09-17T00:33:28.467 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:33:28 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-7[66902]: 2024-09-17T00:33:28.170+0000 7f5a55fab640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-17T00:33:28.467 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:33:28 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-7[66902]: 2024-09-17T00:33:28.170+0000 7f5a55fab640 -1 osd.7 153 *** Got signal Terminated *** 2024-09-17T00:33:28.467 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:33:28 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-7[66902]: 2024-09-17T00:33:28.170+0000 7f5a55fab640 -1 osd.7 153 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-17T00:33:28.467 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:33:28 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-7[66902]: 2024-09-17T00:33:28.194+0000 7f5a4e5b9640 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-09-17T00:32:33.676078+0000 front 2024-09-17T00:32:33.676062+0000 (oldest deadline 2024-09-17T00:32:57.775877+0000) 2024-09-17T00:33:28.467 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:33:28 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-7[66902]: 2024-09-17T00:33:28.194+0000 7f5a4e5b9640 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.2:6814 osd.1 since back 2024-09-17T00:32:44.777292+0000 front 2024-09-17T00:32:44.777224+0000 (oldest deadline 2024-09-17T00:33:08.277122+0000) 2024-09-17T00:33:28.467 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:33:28 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-7[66902]: 2024-09-17T00:33:28.194+0000 7f5a4e5b9640 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.2:6822 osd.2 since back 2024-09-17T00:32:52.377964+0000 front 2024-09-17T00:32:52.378040+0000 (oldest deadline 2024-09-17T00:33:17.677876+0000) 2024-09-17T00:33:28.467 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:33:28 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-7[66902]: 2024-09-17T00:33:28.194+0000 7f5a4e5b9640 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.2:6830 osd.3 since back 2024-09-17T00:33:00.478968+0000 front 2024-09-17T00:33:00.478933+0000 (oldest deadline 2024-09-17T00:33:21.578873+0000) 2024-09-17T00:33:29.467 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:33:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-7[66902]: 2024-09-17T00:33:29.159+0000 7f5a4e5b9640 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-09-17T00:32:33.676078+0000 front 2024-09-17T00:32:33.676062+0000 (oldest deadline 2024-09-17T00:32:57.775877+0000) 2024-09-17T00:33:29.467 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:33:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-7[66902]: 2024-09-17T00:33:29.159+0000 7f5a4e5b9640 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.2:6814 osd.1 since back 2024-09-17T00:32:44.777292+0000 front 2024-09-17T00:32:44.777224+0000 (oldest deadline 2024-09-17T00:33:08.277122+0000) 2024-09-17T00:33:29.467 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:33:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-7[66902]: 2024-09-17T00:33:29.159+0000 7f5a4e5b9640 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.2:6822 osd.2 since back 2024-09-17T00:32:52.377964+0000 front 2024-09-17T00:32:52.378040+0000 (oldest deadline 2024-09-17T00:33:17.677876+0000) 2024-09-17T00:33:29.467 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:33:29 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-7[66902]: 2024-09-17T00:33:29.159+0000 7f5a4e5b9640 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.2:6830 osd.3 since back 2024-09-17T00:33:00.478968+0000 front 2024-09-17T00:33:00.478933+0000 (oldest deadline 2024-09-17T00:33:21.578873+0000) 2024-09-17T00:33:30.217 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:33:30 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-7[66902]: 2024-09-17T00:33:30.150+0000 7f5a4e5b9640 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-09-17T00:32:33.676078+0000 front 2024-09-17T00:32:33.676062+0000 (oldest deadline 2024-09-17T00:32:57.775877+0000) 2024-09-17T00:33:30.217 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:33:30 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-7[66902]: 2024-09-17T00:33:30.150+0000 7f5a4e5b9640 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.2:6814 osd.1 since back 2024-09-17T00:32:44.777292+0000 front 2024-09-17T00:32:44.777224+0000 (oldest deadline 2024-09-17T00:33:08.277122+0000) 2024-09-17T00:33:30.217 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:33:30 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-7[66902]: 2024-09-17T00:33:30.150+0000 7f5a4e5b9640 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.2:6822 osd.2 since back 2024-09-17T00:32:52.377964+0000 front 2024-09-17T00:32:52.378040+0000 (oldest deadline 2024-09-17T00:33:17.677876+0000) 2024-09-17T00:33:30.217 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:33:30 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-7[66902]: 2024-09-17T00:33:30.150+0000 7f5a4e5b9640 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.2:6830 osd.3 since back 2024-09-17T00:33:00.478968+0000 front 2024-09-17T00:33:00.478933+0000 (oldest deadline 2024-09-17T00:33:21.578873+0000) 2024-09-17T00:33:31.467 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:33:31 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-7[66902]: 2024-09-17T00:33:31.185+0000 7f5a4e5b9640 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-09-17T00:32:33.676078+0000 front 2024-09-17T00:32:33.676062+0000 (oldest deadline 2024-09-17T00:32:57.775877+0000) 2024-09-17T00:33:31.467 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:33:31 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-7[66902]: 2024-09-17T00:33:31.185+0000 7f5a4e5b9640 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.2:6814 osd.1 since back 2024-09-17T00:32:44.777292+0000 front 2024-09-17T00:32:44.777224+0000 (oldest deadline 2024-09-17T00:33:08.277122+0000) 2024-09-17T00:33:31.467 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:33:31 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-7[66902]: 2024-09-17T00:33:31.185+0000 7f5a4e5b9640 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.2:6822 osd.2 since back 2024-09-17T00:32:52.377964+0000 front 2024-09-17T00:32:52.378040+0000 (oldest deadline 2024-09-17T00:33:17.677876+0000) 2024-09-17T00:33:31.467 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:33:31 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-7[66902]: 2024-09-17T00:33:31.185+0000 7f5a4e5b9640 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.2:6830 osd.3 since back 2024-09-17T00:33:00.478968+0000 front 2024-09-17T00:33:00.478933+0000 (oldest deadline 2024-09-17T00:33:21.578873+0000) 2024-09-17T00:33:31.467 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:33:31 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-7[66902]: 2024-09-17T00:33:31.185+0000 7f5a4e5b9640 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.99:6804 osd.4 since back 2024-09-17T00:33:04.479602+0000 front 2024-09-17T00:33:04.479624+0000 (oldest deadline 2024-09-17T00:33:30.279891+0000) 2024-09-17T00:33:32.467 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:33:32 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-7[66902]: 2024-09-17T00:33:32.190+0000 7f5a4e5b9640 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.2:6806 osd.0 since back 2024-09-17T00:32:33.676078+0000 front 2024-09-17T00:32:33.676062+0000 (oldest deadline 2024-09-17T00:32:57.775877+0000) 2024-09-17T00:33:32.467 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:33:32 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-7[66902]: 2024-09-17T00:33:32.190+0000 7f5a4e5b9640 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.2:6814 osd.1 since back 2024-09-17T00:32:44.777292+0000 front 2024-09-17T00:32:44.777224+0000 (oldest deadline 2024-09-17T00:33:08.277122+0000) 2024-09-17T00:33:32.467 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:33:32 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-7[66902]: 2024-09-17T00:33:32.190+0000 7f5a4e5b9640 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.2:6822 osd.2 since back 2024-09-17T00:32:52.377964+0000 front 2024-09-17T00:32:52.378040+0000 (oldest deadline 2024-09-17T00:33:17.677876+0000) 2024-09-17T00:33:32.467 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:33:32 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-7[66902]: 2024-09-17T00:33:32.190+0000 7f5a4e5b9640 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.2:6830 osd.3 since back 2024-09-17T00:33:00.478968+0000 front 2024-09-17T00:33:00.478933+0000 (oldest deadline 2024-09-17T00:33:21.578873+0000) 2024-09-17T00:33:32.468 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:33:32 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-7[66902]: 2024-09-17T00:33:32.190+0000 7f5a4e5b9640 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.99:6804 osd.4 since back 2024-09-17T00:33:04.479602+0000 front 2024-09-17T00:33:04.479624+0000 (oldest deadline 2024-09-17T00:33:30.279891+0000) 2024-09-17T00:33:33.871 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:33:33 smithi099 podman[76003]: 2024-09-17 00:33:33.574153105 +0000 UTC m=+5.570066926 container died 9821b8012a7e191d67981f3f04e6e48f8034fcc427cbcd8e8a5ac6030270c3ba (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-7, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240731, ceph=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.license=GPLv2, RELEASE=squid-c2ddf97, maintainer=Guillaume Abrioux ) 2024-09-17T00:33:34.217 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:33:34 smithi099 podman[76003]: 2024-09-17 00:33:34.046491352 +0000 UTC m=+6.042405174 container cleanup 9821b8012a7e191d67981f3f04e6e48f8034fcc427cbcd8e8a5ac6030270c3ba (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-7, org.label-schema.schema-version=1.0, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, RELEASE=squid-c2ddf97, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=) 2024-09-17T00:33:34.217 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:33:34 smithi099 bash[76003]: ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-7 2024-09-17T00:33:34.663 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:33:34 smithi099 podman[76030]: 2024-09-17 00:33:34.306253297 +0000 UTC m=+0.729377460 container remove 9821b8012a7e191d67981f3f04e6e48f8034fcc427cbcd8e8a5ac6030270c3ba (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-7, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-c2ddf97, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, ceph=True, org.label-schema.schema-version=1.0) 2024-09-17T00:33:34.729 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Sep 17 00:33:34 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-alertmanager-a[58000]: ts=2024-09-17T00:33:34.241Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi099.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.99:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi002.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.2:8443: connect: connection refused" 2024-09-17T00:33:34.729 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Sep 17 00:33:34 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-alertmanager-a[58000]: ts=2024-09-17T00:33:34.242Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi002.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.2:8443: connect: connection refused" 2024-09-17T00:33:34.729 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Sep 17 00:33:34 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-alertmanager-a[58000]: ts=2024-09-17T00:33:34.242Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi099.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.99:8443: connect: connection refused" 2024-09-17T00:33:34.967 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:33:34 smithi099 podman[76120]: 2024-09-17 00:33:34.569590023 +0000 UTC m=+0.015983622 image pull ebb0615149def4a3179c82c0cc47af75808300abb9f1142788e5e605a7ac0e5b quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f 2024-09-17T00:33:34.967 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:33:34 smithi099 podman[76120]: 2024-09-17 00:33:34.682059344 +0000 UTC m=+0.128452978 container create 0cd15cd217db57a69f04e80047a73da8f7b035ed265ac76ee0fa563b4152b3e7 (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-7-deactivate, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, GIT_BRANCH=HEAD, ceph=True, RELEASE=squid-c2ddf97, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-09-17T00:33:34.967 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:33:34 smithi099 podman[76120]: 2024-09-17 00:33:34.89291952 +0000 UTC m=+0.339313114 container init 0cd15cd217db57a69f04e80047a73da8f7b035ed265ac76ee0fa563b4152b3e7 (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-7-deactivate, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, RELEASE=squid-c2ddf97, CEPH_POINT_RELEASE=) 2024-09-17T00:33:34.967 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:33:34 smithi099 podman[76120]: 2024-09-17 00:33:34.896218646 +0000 UTC m=+0.342612236 container start 0cd15cd217db57a69f04e80047a73da8f7b035ed265ac76ee0fa563b4152b3e7 (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-7-deactivate, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, RELEASE=squid-c2ddf97, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-09-17T00:33:35.377 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:33:34 smithi099 podman[76120]: 2024-09-17 00:33:34.976042227 +0000 UTC m=+0.422435828 container attach 0cd15cd217db57a69f04e80047a73da8f7b035ed265ac76ee0fa563b4152b3e7 (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-7-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, RELEASE=squid-c2ddf97, ceph=True, GIT_CLEAN=True, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2) 2024-09-17T00:33:35.377 INFO:journalctl@ceph.osd.7.smithi099.stdout:Sep 17 00:33:35 smithi099 podman[76120]: 2024-09-17 00:33:35.085685092 +0000 UTC m=+0.532078699 container died 0cd15cd217db57a69f04e80047a73da8f7b035ed265ac76ee0fa563b4152b3e7 (image=quay.ceph.io/ceph-ci/ceph:c2ddf9721ef254645201cdbabd9c8f49c30a586f, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-osd-7-deactivate, GIT_COMMIT=009c5a8162e3d9e92e49bb850b92bd3cd406d965, RELEASE=squid-c2ddf97, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-09-17T00:33:35.683 DEBUG:teuthology.orchestra.run.smithi099:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3aad27a2-7489-11ef-bceb-c7b262605968@osd.7.service' 2024-09-17T00:33:35.723 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-17T00:33:35.723 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-09-17T00:33:35.723 INFO:tasks.cephadm.prometheus.a:Stopping prometheus.a... 2024-09-17T00:33:35.724 DEBUG:teuthology.orchestra.run.smithi099:> sudo systemctl stop ceph-3aad27a2-7489-11ef-bceb-c7b262605968@prometheus.a 2024-09-17T00:33:35.967 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:33:35 smithi099 systemd[1]: Stopping Ceph prometheus.a for 3aad27a2-7489-11ef-bceb-c7b262605968... 2024-09-17T00:33:36.356 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:33:36 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[52757]: ts=2024-09-17T00:33:36.074Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-09-17T00:33:36.356 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:33:36 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[52757]: ts=2024-09-17T00:33:36.075Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-09-17T00:33:36.356 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:33:36 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[52757]: ts=2024-09-17T00:33:36.075Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-09-17T00:33:36.356 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:33:36 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[52757]: ts=2024-09-17T00:33:36.075Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-09-17T00:33:36.356 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:33:36 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[52757]: ts=2024-09-17T00:33:36.075Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-09-17T00:33:36.356 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:33:36 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[52757]: ts=2024-09-17T00:33:36.075Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-09-17T00:33:36.356 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:33:36 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[52757]: ts=2024-09-17T00:33:36.075Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-09-17T00:33:36.356 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:33:36 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[52757]: ts=2024-09-17T00:33:36.075Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-09-17T00:33:36.356 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:33:36 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[52757]: ts=2024-09-17T00:33:36.075Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-09-17T00:33:36.356 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:33:36 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[52757]: ts=2024-09-17T00:33:36.075Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-09-17T00:33:36.356 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:33:36 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[52757]: ts=2024-09-17T00:33:36.075Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-09-17T00:33:36.357 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:33:36 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a[52757]: ts=2024-09-17T00:33:36.075Z caller=main.go:1147 level=info msg="See you next time!" 2024-09-17T00:33:36.357 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:33:36 smithi099 podman[76223]: 2024-09-17 00:33:36.085361868 +0000 UTC m=+0.135143562 container died bf32c2ae11dc1ea435bf07e899fff851eaeb008d222407034d1335f0c7d22a0b (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-17T00:33:36.706 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:33:36 smithi099 podman[76223]: 2024-09-17 00:33:36.458494358 +0000 UTC m=+0.508276085 container cleanup bf32c2ae11dc1ea435bf07e899fff851eaeb008d222407034d1335f0c7d22a0b (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-17T00:33:36.706 INFO:journalctl@ceph.prometheus.a.smithi099.stdout:Sep 17 00:33:36 smithi099 bash[76223]: ceph-3aad27a2-7489-11ef-bceb-c7b262605968-prometheus-a 2024-09-17T00:33:36.837 DEBUG:teuthology.orchestra.run.smithi099:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3aad27a2-7489-11ef-bceb-c7b262605968@prometheus.a.service' 2024-09-17T00:33:36.871 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-17T00:33:36.871 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a 2024-09-17T00:33:36.871 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 3aad27a2-7489-11ef-bceb-c7b262605968 --force --keep-logs 2024-09-17T00:33:38.732 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Sep 17 00:33:38 smithi002 systemd[1]: Stopping Ceph node-exporter.a for 3aad27a2-7489-11ef-bceb-c7b262605968... 2024-09-17T00:33:39.118 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Sep 17 00:33:38 smithi002 podman[96260]: 2024-09-17 00:33:38.731982843 +0000 UTC m=+0.110136927 container died 1030ae01bb32fb47fb23cd62a696b97e2d3be9b915056ff66a6f4a9ab8e90727 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-17T00:33:39.479 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Sep 17 00:33:39 smithi002 podman[96260]: 2024-09-17 00:33:39.295609932 +0000 UTC m=+0.673764045 container cleanup 1030ae01bb32fb47fb23cd62a696b97e2d3be9b915056ff66a6f4a9ab8e90727 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-17T00:33:39.479 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Sep 17 00:33:39 smithi002 bash[96260]: ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-a 2024-09-17T00:33:39.978 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Sep 17 00:33:39 smithi002 podman[96273]: 2024-09-17 00:33:39.606115097 +0000 UTC m=+0.876288217 container remove 1030ae01bb32fb47fb23cd62a696b97e2d3be9b915056ff66a6f4a9ab8e90727 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-17T00:33:39.979 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Sep 17 00:33:39 smithi002 systemd[1]: ceph-3aad27a2-7489-11ef-bceb-c7b262605968@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-09-17T00:33:39.979 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Sep 17 00:33:39 smithi002 systemd[1]: ceph-3aad27a2-7489-11ef-bceb-c7b262605968@node-exporter.a.service: Failed with result 'exit-code'. 2024-09-17T00:33:39.979 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Sep 17 00:33:39 smithi002 systemd[1]: Stopped Ceph node-exporter.a for 3aad27a2-7489-11ef-bceb-c7b262605968. 2024-09-17T00:33:39.979 INFO:journalctl@ceph.node-exporter.a.smithi002.stdout:Sep 17 00:33:39 smithi002 systemd[1]: ceph-3aad27a2-7489-11ef-bceb-c7b262605968@node-exporter.a.service: Consumed 3.040s CPU time. 2024-09-17T00:33:40.229 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Sep 17 00:33:39 smithi002 systemd[1]: Stopping Ceph alertmanager.a for 3aad27a2-7489-11ef-bceb-c7b262605968... 2024-09-17T00:33:40.229 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Sep 17 00:33:40 smithi002 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-alertmanager-a[58000]: ts=2024-09-17T00:33:40.225Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-09-17T00:33:40.682 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Sep 17 00:33:40 smithi002 podman[96379]: 2024-09-17 00:33:40.266064686 +0000 UTC m=+0.141879161 container died a21e13b3d67719c3fd78a1a24cfb5cd52ae1bbafd1a6624342f9d7028331d94c (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-17T00:33:40.979 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Sep 17 00:33:40 smithi002 podman[96379]: 2024-09-17 00:33:40.792107554 +0000 UTC m=+0.667922028 container cleanup a21e13b3d67719c3fd78a1a24cfb5cd52ae1bbafd1a6624342f9d7028331d94c (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-17T00:33:40.979 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Sep 17 00:33:40 smithi002 bash[96379]: ceph-3aad27a2-7489-11ef-bceb-c7b262605968-alertmanager-a 2024-09-17T00:33:41.369 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Sep 17 00:33:41 smithi002 podman[96391]: 2024-09-17 00:33:41.010077762 +0000 UTC m=+0.735431717 container remove a21e13b3d67719c3fd78a1a24cfb5cd52ae1bbafd1a6624342f9d7028331d94c (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-17T00:33:41.369 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Sep 17 00:33:41 smithi002 podman[96391]: 2024-09-17 00:33:41.085955239 +0000 UTC m=+0.811309150 volume remove 3a9c87d8fad1e17a1537f1b96110a81e6e678e5783bd08b2b078ebed1ef087cd 2024-09-17T00:33:41.729 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Sep 17 00:33:41 smithi002 systemd[1]: ceph-3aad27a2-7489-11ef-bceb-c7b262605968@alertmanager.a.service: Deactivated successfully. 2024-09-17T00:33:41.729 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Sep 17 00:33:41 smithi002 systemd[1]: Stopped Ceph alertmanager.a for 3aad27a2-7489-11ef-bceb-c7b262605968. 2024-09-17T00:33:41.729 INFO:journalctl@ceph.alertmanager.a.smithi002.stdout:Sep 17 00:33:41 smithi002 systemd[1]: ceph-3aad27a2-7489-11ef-bceb-c7b262605968@alertmanager.a.service: Consumed 1.585s CPU time. 2024-09-17T00:34:15.599 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 3aad27a2-7489-11ef-bceb-c7b262605968 --force --keep-logs 2024-09-17T00:34:17.456 INFO:journalctl@ceph.node-exporter.b.smithi099.stdout:Sep 17 00:34:17 smithi099 systemd[1]: Stopping Ceph node-exporter.b for 3aad27a2-7489-11ef-bceb-c7b262605968... 2024-09-17T00:34:17.717 INFO:journalctl@ceph.node-exporter.b.smithi099.stdout:Sep 17 00:34:17 smithi099 podman[76632]: 2024-09-17 00:34:17.458926196 +0000 UTC m=+0.118583808 container died 01fcd35dca8a80afb348bd4c20910055034ea1182396da895bde6f280f96c996 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-17T00:34:18.071 INFO:journalctl@ceph.node-exporter.b.smithi099.stdout:Sep 17 00:34:17 smithi099 podman[76632]: 2024-09-17 00:34:17.881140153 +0000 UTC m=+0.540797752 container cleanup 01fcd35dca8a80afb348bd4c20910055034ea1182396da895bde6f280f96c996 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-17T00:34:18.071 INFO:journalctl@ceph.node-exporter.b.smithi099.stdout:Sep 17 00:34:17 smithi099 bash[76632]: ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-b 2024-09-17T00:34:18.467 INFO:journalctl@ceph.node-exporter.b.smithi099.stdout:Sep 17 00:34:18 smithi099 podman[76648]: 2024-09-17 00:34:18.074154652 +0000 UTC m=+0.615844747 container remove 01fcd35dca8a80afb348bd4c20910055034ea1182396da895bde6f280f96c996 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-17T00:34:18.467 INFO:journalctl@ceph.node-exporter.b.smithi099.stdout:Sep 17 00:34:18 smithi099 systemd[1]: ceph-3aad27a2-7489-11ef-bceb-c7b262605968@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-09-17T00:34:18.467 INFO:journalctl@ceph.node-exporter.b.smithi099.stdout:Sep 17 00:34:18 smithi099 systemd[1]: ceph-3aad27a2-7489-11ef-bceb-c7b262605968@node-exporter.b.service: Failed with result 'exit-code'. 2024-09-17T00:34:18.467 INFO:journalctl@ceph.node-exporter.b.smithi099.stdout:Sep 17 00:34:18 smithi099 systemd[1]: Stopped Ceph node-exporter.b for 3aad27a2-7489-11ef-bceb-c7b262605968. 2024-09-17T00:34:18.468 INFO:journalctl@ceph.node-exporter.b.smithi099.stdout:Sep 17 00:34:18 smithi099 systemd[1]: ceph-3aad27a2-7489-11ef-bceb-c7b262605968@node-exporter.b.service: Consumed 2.801s CPU time. 2024-09-17T00:34:18.866 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:34:18 smithi099 systemd[1]: Stopping Ceph grafana.a for 3aad27a2-7489-11ef-bceb-c7b262605968... 2024-09-17T00:34:18.866 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:34:18 smithi099 ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a[50042]: t=2024-09-17T00:34:18+0000 lvl=info msg="Shutdown started" logger=server reason="System signal: terminated" 2024-09-17T00:34:18.866 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:34:18 smithi099 podman[76726]: 2024-09-17 00:34:18.603994115 +0000 UTC m=+0.111341862 container died bafb824a3c0802109f37ce34f4f387b35333f335f235abda3a6afb6e8e067e00 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a, maintainer=Paul Cuzner , architecture=x86_64, description=Ceph Grafana Container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, build-date=2022-03-28T10:36:18.413762, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, com.redhat.component=ubi8-container, release=236.1648460182, summary=Grafana Container configured for Ceph mgr/dashboard integration, name=ubi8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, vcs-type=git, vendor=Red Hat, Inc., io.openshift.tags=base rhel8, distribution-scope=public, io.k8s.display-name=Red Hat Universal Base Image 8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.24.2, version=8.5, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56) 2024-09-17T00:34:19.217 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:34:18 smithi099 podman[76726]: 2024-09-17 00:34:18.951839218 +0000 UTC m=+0.459186964 container cleanup bafb824a3c0802109f37ce34f4f387b35333f335f235abda3a6afb6e8e067e00 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a, maintainer=Paul Cuzner , architecture=x86_64, io.openshift.expose-services=, vendor=Red Hat, Inc., io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, com.redhat.component=ubi8-container, distribution-scope=public, description=Ceph Grafana Container, version=8.5, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.buildah.version=1.24.2, io.openshift.tags=base rhel8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, vcs-type=git, release=236.1648460182, build-date=2022-03-28T10:36:18.413762, io.k8s.display-name=Red Hat Universal Base Image 8, name=ubi8, summary=Grafana Container configured for Ceph mgr/dashboard integration) 2024-09-17T00:34:19.217 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:34:18 smithi099 bash[76726]: ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a 2024-09-17T00:34:19.217 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:34:18 smithi099 bash[76752]: Error: no container with name or ID "ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana.a" found: no such container 2024-09-17T00:34:19.700 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:34:19 smithi099 podman[76738]: 2024-09-17 00:34:19.244884927 +0000 UTC m=+0.642138505 container remove bafb824a3c0802109f37ce34f4f387b35333f335f235abda3a6afb6e8e067e00 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-3aad27a2-7489-11ef-bceb-c7b262605968-grafana-a, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.openshift.tags=base rhel8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, architecture=x86_64, com.redhat.component=ubi8-container, summary=Grafana Container configured for Ceph mgr/dashboard integration, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, name=ubi8, io.openshift.expose-services=, io.k8s.display-name=Red Hat Universal Base Image 8, io.buildah.version=1.24.2, release=236.1648460182, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Paul Cuzner , vendor=Red Hat, Inc., build-date=2022-03-28T10:36:18.413762, distribution-scope=public, vcs-type=git, version=8.5, description=Ceph Grafana Container) 2024-09-17T00:34:19.700 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:34:19 smithi099 systemd[1]: ceph-3aad27a2-7489-11ef-bceb-c7b262605968@grafana.a.service: Deactivated successfully. 2024-09-17T00:34:19.700 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:34:19 smithi099 systemd[1]: Stopped Ceph grafana.a for 3aad27a2-7489-11ef-bceb-c7b262605968. 2024-09-17T00:34:19.700 INFO:journalctl@ceph.grafana.a.smithi099.stdout:Sep 17 00:34:19 smithi099 systemd[1]: ceph-3aad27a2-7489-11ef-bceb-c7b262605968@grafana.a.service: Consumed 4.173s CPU time. 2024-09-17T00:34:42.306 DEBUG:teuthology.orchestra.run.smithi002:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-17T00:34:42.341 DEBUG:teuthology.orchestra.run.smithi099:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-17T00:34:42.368 INFO:tasks.cephadm:Archiving crash dumps... 2024-09-17T00:34:42.369 DEBUG:teuthology.misc:Transferring archived files from smithi002:/var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-09-16_21:08:04-orch-squid-distro-default-smithi/7907924/remote/smithi002/crash 2024-09-17T00:34:42.370 DEBUG:teuthology.orchestra.run.smithi002:> sudo tar c -f - -C /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/crash -- . 2024-09-17T00:34:42.429 INFO:teuthology.orchestra.run.smithi002.stderr:tar: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/crash: Cannot open: No such file or directory 2024-09-17T00:34:42.430 INFO:teuthology.orchestra.run.smithi002.stderr:tar: Error is not recoverable: exiting now 2024-09-17T00:34:42.431 DEBUG:teuthology.misc:Transferring archived files from smithi099:/var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-09-16_21:08:04-orch-squid-distro-default-smithi/7907924/remote/smithi099/crash 2024-09-17T00:34:42.432 DEBUG:teuthology.orchestra.run.smithi099:> sudo tar c -f - -C /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/crash -- . 2024-09-17T00:34:42.474 INFO:teuthology.orchestra.run.smithi099.stderr:tar: /var/lib/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/crash: Cannot open: No such file or directory 2024-09-17T00:34:42.474 INFO:teuthology.orchestra.run.smithi099.stderr:tar: Error is not recoverable: exiting now 2024-09-17T00:34:42.475 INFO:tasks.cephadm:Checking cluster log for badness... 2024-09-17T00:34:42.475 DEBUG:teuthology.orchestra.run.smithi002:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/ceph.log | egrep CEPHADM_ | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v CEPHADM_STRAY_DAEMON | egrep -v CEPHADM_FAILED_DAEMON | egrep -v CEPHADM_AGENT_DOWN | head -n 1 2024-09-17T00:34:42.509 INFO:tasks.cephadm:Compressing logs... 2024-09-17T00:34:42.509 DEBUG:teuthology.orchestra.run.smithi002:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-17T00:34:42.552 DEBUG:teuthology.orchestra.run.smithi099:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-17T00:34:42.578 INFO:teuthology.orchestra.run.smithi099.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/ceph-osd.7.log 2024-09-17T00:34:42.578 INFO:teuthology.orchestra.run.smithi099.stderr:: No such file or directory 2024-09-17T00:34:42.579 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/ceph-volume.log 2024-09-17T00:34:42.579 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/ceph-osd.4.log 2024-09-17T00:34:42.579 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/ceph-osd.7.log: /var/log/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/ceph.audit.log 2024-09-17T00:34:42.579 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/ceph.cephadm.log 2024-09-17T00:34:42.579 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/ceph-osd.6.log 2024-09-17T00:34:42.579 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/ceph-osd.5.log 2024-09-17T00:34:42.579 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/ceph.log 2024-09-17T00:34:42.579 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/ceph-mgr.x.log 2024-09-17T00:34:42.579 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/ceph.log: 92.1% -- replaced with /var/log/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/ceph.cephadm.log.gz 2024-09-17T00:34:42.582 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/ceph-client.rgw.foo.smithi099.utzfap.log 2024-09-17T00:34:42.582 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/ceph-mgr.x.log: 87.6% -- replaced with /var/log/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/ceph.log.gz 2024-09-17T00:34:42.582 INFO:teuthology.orchestra.run.smithi099.stderr: 90.5% -- replaced with /var/log/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/ceph.audit.log.gz 2024-09-17T00:34:42.585 INFO:teuthology.orchestra.run.smithi002.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/ceph-osd.0.log 2024-09-17T00:34:42.585 INFO:teuthology.orchestra.run.smithi002.stderr:: No such file or directory 2024-09-17T00:34:42.586 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/tcmu-runner.log 2024-09-17T00:34:42.586 INFO:teuthology.orchestra.run.smithi002.stderr:gzip/var/log/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/ceph-osd.0.log: -5 --verbose -- /var/log/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/ceph-osd.1.log 2024-09-17T00:34:42.586 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/tcmu-runner.log: gzip -5 --verbose -- /var/log/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/ceph-volume.log 2024-09-17T00:34:42.586 INFO:teuthology.orchestra.run.smithi002.stderr: 90.5% -- replaced with /var/log/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/tcmu-runner.log.gz 2024-09-17T00:34:42.586 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/ceph.audit.log 2024-09-17T00:34:42.586 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/ceph-client.rgw.smpl.smithi002.rqnxii.log 2024-09-17T00:34:42.586 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/ceph-mon.c.log 2024-09-17T00:34:42.586 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/ceph-client.rgw.smpl.smithi002.rqnxii.log: gzip -5 --verbose -- /var/log/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/ceph.cephadm.log 2024-09-17T00:34:42.586 INFO:teuthology.orchestra.run.smithi002.stderr: 76.3% -- replaced with /var/log/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/ceph-client.rgw.smpl.smithi002.rqnxii.log.gz/var/log/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/ceph-mon.c.log: 2024-09-17T00:34:42.586 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/ceph-mon.a.log 2024-09-17T00:34:42.586 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/ceph-osd.2.log 2024-09-17T00:34:42.586 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/ceph-client.rgw.foo.smithi002.uwdhcs.log 2024-09-17T00:34:42.586 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/ceph-osd.2.log: 95.2% -- replaced with /var/log/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/ceph.cephadm.log.gz 2024-09-17T00:34:42.587 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/ceph-mgr.y.log 2024-09-17T00:34:42.587 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/ceph-client.rgw.foo.smithi002.uwdhcs.log: 76.3% -- replaced with /var/log/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/ceph-client.rgw.foo.smithi002.uwdhcs.log.gz 2024-09-17T00:34:42.587 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/ceph.log 2024-09-17T00:34:42.587 INFO:teuthology.orchestra.run.smithi099.stderr: 91.2% -- replaced with /var/log/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/ceph-volume.log.gz 2024-09-17T00:34:42.588 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/ceph-mgr.y.log: 94.2% -- replaced with /var/log/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/ceph.audit.log.gz 2024-09-17T00:34:42.588 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/ceph-osd.3.log 2024-09-17T00:34:42.591 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/ceph.log: 91.0% -- replaced with /var/log/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/ceph-volume.log.gz 2024-09-17T00:34:42.592 INFO:teuthology.orchestra.run.smithi002.stderr: 93.1% -- replaced with /var/log/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/ceph.log.gz 2024-09-17T00:34:42.593 INFO:teuthology.orchestra.run.smithi002.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-09-17T00:34:42.594 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/ceph-mon.b.log 2024-09-17T00:34:42.596 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/ceph-client.rgw.foo.smithi099.utzfap.log: gzip -5 --verbose -- /var/log/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/ceph-client.rgw.smpl.smithi099.nzvkpm.log 2024-09-17T00:34:42.597 INFO:teuthology.orchestra.run.smithi099.stderr: 76.6% -- replaced with /var/log/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/ceph-client.rgw.foo.smithi099.utzfap.log.gz/var/log/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/ceph-mon.b.log: 2024-09-17T00:34:42.597 INFO:teuthology.orchestra.run.smithi099.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-09-17T00:34:42.597 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/ceph-client.rgw.smpl.smithi099.nzvkpm.log: 76.4% -- replaced with /var/log/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/ceph-client.rgw.smpl.smithi099.nzvkpm.log.gz 2024-09-17T00:34:42.606 INFO:teuthology.orchestra.run.smithi002.stderr:/var/log/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/ceph-osd.3.log: /var/log/ceph/cephadm.log: 91.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-09-17T00:34:42.616 INFO:teuthology.orchestra.run.smithi099.stderr:/var/log/ceph/cephadm.log: 90.8% -- replaced with /var/log/ceph/cephadm.log.gz 2024-09-17T00:34:42.647 INFO:teuthology.orchestra.run.smithi099.stderr: 89.7% -- replaced with /var/log/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/ceph-mgr.x.log.gz 2024-09-17T00:34:42.907 INFO:teuthology.orchestra.run.smithi099.stderr: 92.3% -- replaced with /var/log/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/ceph-mon.b.log.gz 2024-09-17T00:34:42.956 INFO:teuthology.orchestra.run.smithi002.stderr: 88.9% -- replaced with /var/log/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/ceph-mgr.y.log.gz 2024-09-17T00:34:42.962 INFO:teuthology.orchestra.run.smithi002.stderr: 92.1% -- replaced with /var/log/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/ceph-mon.c.log.gz 2024-09-17T00:34:43.530 INFO:teuthology.orchestra.run.smithi002.stderr: 93.6% -- replaced with /var/log/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/ceph-osd.2.log.gz 2024-09-17T00:34:43.694 INFO:teuthology.orchestra.run.smithi099.stderr: 93.5% -- replaced with /var/log/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/ceph-osd.6.log.gz 2024-09-17T00:34:43.740 INFO:teuthology.orchestra.run.smithi002.stderr: 90.9% -- replaced with /var/log/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/ceph-mon.a.log.gz 2024-09-17T00:34:44.025 INFO:teuthology.orchestra.run.smithi099.stderr: 94.0% -- replaced with /var/log/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/ceph-osd.7.log.gz 2024-09-17T00:34:44.054 INFO:teuthology.orchestra.run.smithi099.stderr: 93.6% -- replaced with /var/log/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/ceph-osd.5.log.gz 2024-09-17T00:34:44.205 INFO:teuthology.orchestra.run.smithi099.stderr: 93.8% -- replaced with /var/log/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/ceph-osd.4.log.gz 2024-09-17T00:34:44.207 INFO:teuthology.orchestra.run.smithi099.stderr: 2024-09-17T00:34:44.207 INFO:teuthology.orchestra.run.smithi099.stderr:real 0m1.641s 2024-09-17T00:34:44.207 INFO:teuthology.orchestra.run.smithi099.stderr:user 0m5.856s 2024-09-17T00:34:44.207 INFO:teuthology.orchestra.run.smithi099.stderr:sys 0m0.251s 2024-09-17T00:34:44.221 INFO:teuthology.orchestra.run.smithi002.stderr: 93.7% -- replaced with /var/log/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/ceph-osd.0.log.gz 2024-09-17T00:34:44.246 INFO:teuthology.orchestra.run.smithi002.stderr: 93.7% -- replaced with /var/log/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/ceph-osd.1.log.gz 2024-09-17T00:34:44.385 INFO:teuthology.orchestra.run.smithi002.stderr: 93.7% -- replaced with /var/log/ceph/3aad27a2-7489-11ef-bceb-c7b262605968/ceph-osd.3.log.gz 2024-09-17T00:34:44.387 INFO:teuthology.orchestra.run.smithi002.stderr: 2024-09-17T00:34:44.387 INFO:teuthology.orchestra.run.smithi002.stderr:real 0m1.820s 2024-09-17T00:34:44.387 INFO:teuthology.orchestra.run.smithi002.stderr:user 0m7.751s 2024-09-17T00:34:44.387 INFO:teuthology.orchestra.run.smithi002.stderr:sys 0m0.268s 2024-09-17T00:34:44.387 INFO:tasks.cephadm:Archiving logs... 2024-09-17T00:34:44.388 DEBUG:teuthology.misc:Transferring archived files from smithi002:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-16_21:08:04-orch-squid-distro-default-smithi/7907924/remote/smithi002/log 2024-09-17T00:34:44.389 DEBUG:teuthology.orchestra.run.smithi002:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-17T00:34:45.431 DEBUG:teuthology.misc:Transferring archived files from smithi099:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-16_21:08:04-orch-squid-distro-default-smithi/7907924/remote/smithi099/log 2024-09-17T00:34:45.432 DEBUG:teuthology.orchestra.run.smithi099:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-17T00:34:46.397 INFO:tasks.cephadm:Removing cluster... 2024-09-17T00:34:46.397 DEBUG:teuthology.orchestra.run.smithi002:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 3aad27a2-7489-11ef-bceb-c7b262605968 --force 2024-09-17T00:34:46.693 DEBUG:teuthology.orchestra.run.smithi099:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 3aad27a2-7489-11ef-bceb-c7b262605968 --force 2024-09-17T00:34:46.978 INFO:tasks.cephadm:Removing cephadm ... 2024-09-17T00:34:47.115 DEBUG:teuthology.orchestra.run.smithi002:> rm -rf /home/ubuntu/cephtest/cephadm 2024-09-17T00:34:47.132 DEBUG:teuthology.orchestra.run.smithi099:> rm -rf /home/ubuntu/cephtest/cephadm 2024-09-17T00:34:47.148 INFO:tasks.cephadm:Teardown complete 2024-09-17T00:34:47.148 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-17T00:34:47.158 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-17T00:34:47.158 DEBUG:teuthology.orchestra.run.smithi002:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-17T00:34:47.174 DEBUG:teuthology.orchestra.run.smithi099:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-17T00:34:47.188 INFO:teuthology.orchestra.run.smithi002.stderr:bash: line 1: ntpq: command not found 2024-09-17T00:34:47.192 INFO:teuthology.orchestra.run.smithi002.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-17T00:34:47.192 INFO:teuthology.orchestra.run.smithi002.stdout:=============================================================================== 2024-09-17T00:34:47.192 INFO:teuthology.orchestra.run.smithi002.stdout:^* hv01.front.sepia.ceph.com 3 6 377 38 +3789us[+3821us] +/- 48ms 2024-09-17T00:34:47.192 INFO:teuthology.orchestra.run.smithi002.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 36 -3773us[-3773us] +/- 63ms 2024-09-17T00:34:47.192 INFO:teuthology.orchestra.run.smithi002.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 37 -1920us[-1920us] +/- 83ms 2024-09-17T00:34:47.192 INFO:teuthology.orchestra.run.smithi002.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T00:34:47.202 INFO:teuthology.orchestra.run.smithi099.stderr:bash: line 1: ntpq: command not found 2024-09-17T00:34:47.206 INFO:teuthology.orchestra.run.smithi099.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-17T00:34:47.206 INFO:teuthology.orchestra.run.smithi099.stdout:=============================================================================== 2024-09-17T00:34:47.206 INFO:teuthology.orchestra.run.smithi099.stdout:^* hv01.front.sepia.ceph.com 3 6 377 38 +3834us[+3685us] +/- 48ms 2024-09-17T00:34:47.206 INFO:teuthology.orchestra.run.smithi099.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 37 -3731us[-3731us] +/- 63ms 2024-09-17T00:34:47.206 INFO:teuthology.orchestra.run.smithi099.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 34 -1886us[-1886us] +/- 83ms 2024-09-17T00:34:47.206 INFO:teuthology.orchestra.run.smithi099.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-09-17T00:34:47.206 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-17T00:34:47.215 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-17T00:34:47.216 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-17T00:34:47.224 DEBUG:teuthology.orchestra.run.smithi002:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-17T00:34:47.318 DEBUG:teuthology.orchestra.run.smithi099:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-17T00:34:47.392 DEBUG:teuthology.orchestra.run.smithi002:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-17T00:34:47.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T00:34:47.422 DEBUG:teuthology.orchestra.run.smithi099:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-17T00:34:47.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T00:34:47.450 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-17T00:34:47.459 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-17T00:34:47.477 INFO:teuthology.task.internal:Duration was 1695.214617 seconds 2024-09-17T00:34:47.477 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-17T00:34:47.490 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-17T00:34:47.491 DEBUG:teuthology.orchestra.run.smithi002:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-17T00:34:47.493 DEBUG:teuthology.orchestra.run.smithi099:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-17T00:34:47.531 INFO:teuthology.orchestra.run.smithi099.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-17T00:34:47.533 INFO:teuthology.orchestra.run.smithi002.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-17T00:34:48.022 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-17T00:34:48.023 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi002.front.sepia.ceph.com 2024-09-17T00:34:48.023 DEBUG:teuthology.orchestra.run.smithi002:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-17T00:34:48.085 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi099.front.sepia.ceph.com 2024-09-17T00:34:48.086 DEBUG:teuthology.orchestra.run.smithi099:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-17T00:34:48.115 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-17T00:34:48.116 DEBUG:teuthology.orchestra.run.smithi002:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-17T00:34:48.127 DEBUG:teuthology.orchestra.run.smithi099:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-17T00:34:48.192 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-17T00:34:48.192 DEBUG:teuthology.orchestra.run.smithi002:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-17T00:34:48.200 DEBUG:teuthology.orchestra.run.smithi099:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-17T00:34:48.673 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-17T00:34:48.682 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-17T00:34:48.683 DEBUG:teuthology.orchestra.run.smithi002:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-17T00:34:48.708 DEBUG:teuthology.orchestra.run.smithi099:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-17T00:34:48.733 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-17T00:34:48.745 DEBUG:teuthology.orchestra.run.smithi002:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-17T00:34:48.750 DEBUG:teuthology.orchestra.run.smithi099:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-17T00:34:48.772 INFO:teuthology.orchestra.run.smithi002.stdout:kernel.core_pattern = core 2024-09-17T00:34:48.798 INFO:teuthology.orchestra.run.smithi099.stdout:kernel.core_pattern = core 2024-09-17T00:34:48.816 DEBUG:teuthology.orchestra.run.smithi002:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-17T00:34:48.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T00:34:48.858 DEBUG:teuthology.orchestra.run.smithi099:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-17T00:34:48.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-17T00:34:48.872 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-17T00:34:48.882 INFO:teuthology.task.internal:Transferring archived files... 2024-09-17T00:34:48.883 DEBUG:teuthology.misc:Transferring archived files from smithi002:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-16_21:08:04-orch-squid-distro-default-smithi/7907924/remote/smithi002 2024-09-17T00:34:48.883 DEBUG:teuthology.orchestra.run.smithi002:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-17T00:34:48.945 DEBUG:teuthology.misc:Transferring archived files from smithi099:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-16_21:08:04-orch-squid-distro-default-smithi/7907924/remote/smithi099 2024-09-17T00:34:48.945 DEBUG:teuthology.orchestra.run.smithi099:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-17T00:34:48.984 INFO:teuthology.task.internal:Removing archive directory... 2024-09-17T00:34:48.984 DEBUG:teuthology.orchestra.run.smithi002:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-17T00:34:48.986 DEBUG:teuthology.orchestra.run.smithi099:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-17T00:34:49.032 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-17T00:34:49.043 INFO:teuthology.task.internal:Not uploading archives. 2024-09-17T00:34:49.043 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-17T00:34:49.056 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-17T00:34:49.057 DEBUG:teuthology.orchestra.run.smithi002:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-17T00:34:49.059 DEBUG:teuthology.orchestra.run.smithi099:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-17T00:34:49.072 INFO:teuthology.orchestra.run.smithi002.stdout: 262169 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 17 00:34 /home/ubuntu/cephtest 2024-09-17T00:34:49.087 INFO:teuthology.orchestra.run.smithi099.stdout: 265324 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 17 00:34 /home/ubuntu/cephtest 2024-09-17T00:34:49.088 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-17T00:34:49.097 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-17T00:34:49.128 INFO:teuthology.run:Summary data: description: orch/cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/classic} duration: 1695.2146165370941 owner: scheduled_teuthology@teuthology success: true 2024-09-17T00:34:49.128 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-17T00:34:49.206 INFO:teuthology.run:pass